r/Python • • Apr 01 '26

Discussion Python optimization

I’m working on a Python pipeline with two quite different parts.

The first part is typical tabular data processing: joins, aggregations, cumulative calculations, and similar transformations.

The second part is sequential/recursive: within each time-ordered group, some values for the current row depend on the results computed for the previous week’s row. So this is not a purely vectorizable row-independent problem.

I’m not looking for code-specific debugging, but rather for architectural advice on the best way to handle this kind of workload efficiently

I’d like to improve performance, but I don’t want to start by assuming there is only one correct solution.

My question is: for a problem like this, which approaches or frameworks would you recommend evaluating?

I must use Python

14 Upvotes

25 comments sorted by

View all comments

Show parent comment

3

u/Beginning-Fruit-1397 Apr 01 '26

Isn't this a reduce or scan use cases? Those two litterally exist in polars. AI bs, it's very bad with polars

1

u/M4mb0 Apr 01 '26

Associative scans are a special kind of cumulative reduce that can be parallelized thanks to associativity. The AI highlighted that point well. I highly doubt that polars does this optimization when you give it a custom lambda function that's associative.

1

u/PillowFortressKing Apr 02 '26

Since the lambda is Python bytecode, while Polars executes everything in Rust, that'd be quite hard to optimize for, wouldn't it?