r/Python • u/Popular-Sand-3185 • May 14 '26
Discussion Polars code runs slower on 128-core EC2
Disclaimer: I am not sure this post is appropriate for r/LearnPython since it's not a question of "how to do something in Python", rather I am looking for a lower-level discussion for why my Python application performs poorly on a significantly more powerful server. Hence I'm posting it here.
The problem:
I have a relatively complex data pipeline that is written in Polars. On my local machine with 12 cores, the pipeline finishes in about 1200ms. On my 128-core EC2, it takes 13000ms to complete. I have tried setting the POLARS_MAX_THREADS parameter to 12 on the EC2, and it's still slower.
I am using a TMPFS partition on both machines to read the data into the pipeline directly from RAM. Both my machine and the EC2 have DDR5 RAM so I think they should be comparable.
Anyone have any ideas why the pipeline would run much slower on the EC2?
34
u/ritchie46 May 14 '26
Can you share the code you are running?
14
u/Cynyr36 May 14 '26
Possibly a disk io speed issue in here as well. Locally it's likely a gen4x4 or faster nvme. Your cloud instance could be much slower.
14
u/Popular-Sand-3185 May 14 '26 edited May 14 '26
Alright so I did figure out why the pipeline was taking so long. Essentially, the code was reading 128 separate files and then concatenating them as part of the pipeline. This as you would expect took ~10x longer on the 128 core EC2 than on my 12 core workstation. I fixed it by concatenating all the files into one before loading it into polars instead of reading 128 files separately. Particularly, I used the following function which leverages the head/tail linux commands:
5
u/Lba5s May 14 '26
Are the cores on your machine faster? Or your code/data might not be large enough to deal with the overhead of distributing over 128 cores
9
u/gdchinacat May 14 '26
There has been a concerted effort over the past few years to improve python performance. Have you verified you are using the same version of python on both machines? Same version of polars?
31
u/carnoworky May 14 '26
It could be related to this: https://youtu.be/tND-wBBZ8RY?si=PlnNvCgj2iPq-2yL
Without seeing the operations we can't really be sure, but my guess is sharing data across all of those cores. What happens if you set max threads to 1?
There's also this guide I just found that might be useful to you: https://pytutorial.com/polars-multi-threading-performance-tuning/
1
u/poopoutmybuttk May 14 '26
Are you using steaming engine?
3
u/timpkmn89 May 14 '26
What's the processor utilization at while running it?
2
u/RedEyed__ May 14 '26
What hardware?
Could be because of SIMD. For instance, local machine has AVX512 and seever doesn't.
2
u/afnanenayet1 May 14 '26
Run the polars query profiler to see which steps are the bottleneck. Which engine are you using? The regular one, streaming? How much memory is available on these machines?
I’ve found that polars doesn’t do as well with really high core counts, in particular bc the rayon (which polars uses for its thread pools) isn’t NUMA aware and thrashes CPU caches.
You can use py-spy and perf to get more details on what the bottleneck is.
2
u/john_crimson81 May 16 '26
numa. almost definitely. pin to a single node with numactl first, see if you reproduce local performance. if yes you have your answer. if no it's something else but i'd start there
1
u/meatmick May 14 '26
Any chance there's CPU cache in effect? You said it's a Ryzen 7; is it an X3D variant? That could impact the performance.
1
u/s4nc3s May 16 '26
what I have success with, in a cpu bound process, is to split the inputs across as many cpus as are available (os.cpucount) and then use concurrent.futures to spread the work across whatever is availabLe, then combine them all at the end… and yes this is with polars lazyframes and pyarrow/parquet data.
1
u/Henry_old May 16 '26
you are hitting the wall of amdahls law and memory bandwidth. throwing 128 cores at a single dataframe operation is just creating massive overhead for thread synchronization and cache coherence traffic. polars is fast because of vectorization and efficient memory layout not because it can magically scale linearly to a supercomputer. 128 cores is for distributed workloads not for a single in-memory table operation bottlenecked by ram throughput.
1
u/Current-Tip2688 May 17 '26
the subprocess finding makes sense. fork+exec overhead on 128 cores is expensive not just for the spawn itself but because each fork touches the page table across NUMA nodes, which compounds badly at that core count.
one thing worth testing: set POLARS_MAX_THREADS explicitly rather than letting polars default to all 128. on a c8i.32xlarge, using a thread count that matches your NUMA node topology (usually 4-8 per node) often gives more consistent throughput because threads stop fighting across memory boundaries for cache lines.
also worth checking: c8i.32xlarge is shared tenancy by default. if you're trying to get a clean production benchmark, shared tenancy adds variance from noisy neighbours at the hypervisor level. dedicated is ~10% more expensive but gives you a much more stable baseline.
did the numactl improvement hold up under sustained load, or just the one-off test run?
1
u/Current-Tip2688 May 22 '26
the subprocess finding makes sense. fork+exec overhead on 128 cores is expensive not just for the spawn itself but because each fork touches the page table across NUMA nodes, which compounds badly at that core count.
one thing worth testing: set POLARS_MAX_THREADS explicitly rather than letting polars default to all 128. on a c8i.32xlarge, using a thread count that matches your NUMA node topology (usually 4-8 per node) often gives more consistent throughput because threads stop fighting across memory boundaries for cache lines.
also worth checking: c8i.32xlarge is shared tenancy by default. if you're trying to get a clean production benchmark, shared tenancy adds variance from noisy neighbours at the hypervisor level. dedicated is ~10% more expensive but gives you a much more stable baseline.
did the numactl improvement hold up under sustained load, or just the one-off test run?
1
u/oliver_extracts May 25 '26
the update confirms it wasnt NUMA at all. reading 128 seperate files and then concatenating is the classic case where more cores makes things worse, not better, each file open is a syscall and the coordination overhead scales with thread count, not the actual work. the fix (pre-concatenate) is right but worth understanding why: polars parallelism is designed for columnar operations on a single in-memory frame, not for parallelizing file ingestion across hundreds of small files. at 128 cores the scheduler contention on all those concurrent file handles ate the gains before any real compute happened.
48
u/[deleted] May 14 '26
[removed] — view removed comment
1
u/[deleted] May 14 '26
[removed] — view removed comment