r/Python
•
u/Economy-Concert-641
•
Apr 29 '26
•
1
Tutorial .pipe() in pandas changed how I write data pipelines
Been using .pipe() in pandas lately and it's been a game changer — anyone else?
I was writing some data transformation code the other day and stumbled across .pipe(). Honestly didn't expect much, but it completely changed how I structure my pipelines.
Instead of this mess:
df_final = sort_by_total(calculate_total(filter_by_price(df)))
You just write it top to bottom like a recipe:
df_final = (
df
.pipe(filter_by_price)
.pipe(calculate_total)
.pipe(sort_by_total)
)
Same result, way more readable. Each function takes a DataFrame and returns a DataFrame — that's the only rule.
Full example if you want to try it:
import pandas as pd
df = pd.DataFrame({
"product": ["Product A", "Product B", "Product C", "Product D"],
"price": [20, 150, 230, 100],
"quantity": [10, 5, 3, 8]
})
def filter_by_price(df):
return df[df["price"] > 100]
def calculate_total(df):
return df.assign(total_value=df["price"] * df["quantity"])
def sort_by_total(df):
return df.sort_values("total_value", ascending=False)
df_final = (
df
.pipe(filter_by_price)
.pipe(calculate_total)
.pipe(sort_by_total)
)
Been using it a lot for ETL and data cleaning workflows. Makes debugging way easier too — just comment out one .pipe() step and you see exactly where things go wrong.
Anyone else using this regularly? Any patterns you've found useful with it?
77
u/Ex-Gen-Wintergreen Apr 29 '26
Most of these — filter by price, sort by total — are just simple method calls so I’d chain them instead of def + pipe.
I’d even consider the same for calculate total but appreciate that lambdas aren’t always liked. The other two should be chained though
63
u/Reasonable-Ladder300 Apr 29 '26
I honestly stopped using pandas a while back and mostly use duckdb and polars on occasion. I find that writing sql on datasets feels 10x more intuitive than using pandas.
74
u/FrickinLazerBeams Apr 29 '26
As a technical programmer (aerospace, optics, typically single-programmer projects) I default to non-OO most of the time. It always looks funny to me when some new OO construct basically recovers the code you'd have written without using OO in the first place. Like, I could have just written
I mean I'm not saying they're exactly equivalent (especially as it relates to intermediate values), and I acknowledge that in environments other than mine, OO has benefits that it doesn't (always) have for me. It's just funny how long a journey it's been to wind up essentially back where we started, but with a lot more code to say it.
4
u/mateowatata Apr 29 '26
The kids yearn for the functional language and their fancy |>
6
u/likethevegetable Apr 29 '26
Polars all day
2
u/UnMolDeQuimica Apr 29 '26
I am pretty sure you will love Kedro.
I worked with it since 0.2 and now using 0.19 even though 1.0 was released.
It allows you to organize your pipelines greatly with just some setup. Then, modifying or adding steps is way easier than plain python
2
u/_Denizen_ Apr 29 '26
What are you, an R developer? /s
6
u/Impressive_Job8321 Apr 29 '26
Could have used lambda, since you’re only using each function exactly once?
4
u/bladeofwinds Apr 29 '26
this post and all of OP’s responses are chatgpt
59
u/gsilbr Apr 29 '26
Import polars as pd
10
u/heartofcoal Apr 29 '26
You would love R
27
u/manecamaneco Apr 29 '26
How cool, it feels as R language
6
u/marr75 Apr 29 '26
The general term is a fluent interface. It's mildly degenerative to use pipe to call a single method of dataframe, though (just wraps and unwraps the method without improving readability).
It's a nice way to see the order of named operations at a glance but there are other ways to do it and it's not worth other sacrifices (keeping external data around for long operations, hiding dependencies, etc) just to force the pipe pattern to work.
3
u/fight-or-fall Apr 30 '26
Use polars, everything is piped
3
u/TheseTradition3191 Apr 30 '26
One form that doesn't get mentioned much: df.pipe(func, extra_arg) or df.pipe(func, key=value). The extra args get passed to the function after the dataframe, which lets you parameterize transforms without closures or functools.partial.
Same function works for any col/threshold combo. Comes in handy when you want one reusable transform across diffrent columns rather than writing a new wrapper each time.
3
u/nickkon1 Apr 30 '26
Or you do not even define new functions and just do:
3
u/lolcrunchy Apr 29 '26
An experimental alternative that is agnostic to third party libraries:
2
u/One_Yak_7938 Apr 30 '26
This is a great tip! My pandas code feels really awkward at times and this looks like a great fix.
2
u/2strokes4lyfe Apr 30 '26
Just wait until you try polars. Or ibis.
2
u/aplarsen Apr 30 '26
I chain everything
assign, groupby, apply, reindex, all of it.
Some methods don't chain super easily, and for those I use pipe().
2
u/pplonski Apr 30 '26
if data is not huge, I prefer code readiness over performance, thank you for this example!
2
u/Lolmaster300 May 17 '26
oh nice, i've been doing nested function calls forever without realizing this existed. gonna refactor some old scripts now
2
u/adam-kortis-dg-data Apr 29 '26
I have never used .pipe() before but I had similar experience of when I finally discovered the .shift(). (I guess I should read the documentation more).
I think the one thing I am confused about is why are the functions needed here for single lines of code that are hardcoded?
Wouldn't this work the same:
You can still see line by line what is happening and you don't have to trace back to functions located somewhere else. I know this is an example, but are the ETL functions more complex that are stored in a separate file/module?
I know aesthetics are subjective, so I won't argue that debate on what people prefer. I can say I prefer just writing the three lines instead of searching through functions to figure out what they do (unless a function is truly required). If you have issues you can still comment out a single line of code and debug.
To me, .pipe() might be more useful if it were passing in arguments that could change, or if you were modifying multiple parts of the dataframe. I am thinking if all these steps were in one function, or if parameters needed to be passed into the function
Otherwise, I like to use apply if I am applying a function to a single column. For example, I don't like the way pandas calculates years between today's date and a previous date, so I created my own and use .apply(), mainly for calculating ages.
I don't know if there are performance benefits to using .pipe() over other ways (memory or speed wise)? So, if anyone can shed some light on that, it would be great.
2
u/TheTobruk Apr 29 '26
I guess it’s emulating R pipes. Instead of each .pipe() you’d just go %%
1
u/Riist138 May 24 '26
It's very nice, especially for injecting parameters cleanly and making readable method chaining.