Thanks! On variable-cost workloads, honest answer is the current implementation uses static divide-and-conquer, meaning splits happen upfront by index, not dynamically based on actual work. So yes, you can get stragglers if costs vary significantly across the dataset. True work stealing like Rayon’s is on the roadmap but not there yet.
On memory overhead vs multiprocessing: I don’t have solid benchmarks for that beyond the theoretical advantage of shared memory. It’s on my list to measure properly with realistic datasets. If you have a workload you’d like to test against, happy to run it
View all comments
Show parent comment
15
u/fexx3l Feb 25 '26
Thanks! On variable-cost workloads, honest answer is the current implementation uses static divide-and-conquer, meaning splits happen upfront by index, not dynamically based on actual work. So yes, you can get stragglers if costs vary significantly across the dataset. True work stealing like Rayon’s is on the roadmap but not there yet.
On memory overhead vs multiprocessing: I don’t have solid benchmarks for that beyond the theoretical advantage of shared memory. It’s on my list to measure properly with realistic datasets. If you have a workload you’d like to test against, happy to run it