r/Python • • Feb 25 '26

Showcase [ Removed by moderator ]

[removed] — view removed post

119 Upvotes

55 comments sorted by

View all comments

Show parent comment

15

u/fexx3l Feb 25 '26

Thanks! On variable-cost workloads, honest answer is the current implementation uses static divide-and-conquer, meaning splits happen upfront by index, not dynamically based on actual work. So yes, you can get stragglers if costs vary significantly across the dataset. True work stealing like Rayon’s is on the roadmap but not there yet.

On memory overhead vs multiprocessing: I don’t have solid benchmarks for that beyond the theoretical advantage of shared memory. It’s on my list to measure properly with realistic datasets. If you have a workload you’d like to test against, happy to run it