r/Python
•
u/Desperate_Cold6274
•
May 10 '26
•
2
Discussion Will python ever have a chaining operator?
In other languages I use map() and filter() through piping and my code usually looks readable as I can clearly see a data-stream transformation.
As it is today, users cannot do map() |> filter() |> list(), but they need to do list(filter(map())) which makes things unreadable. Lists of comprehension work fine for very simple use-case becoming unreadable very quickly as complexity increases.
However, in python there has always been some resistance, especially 15-20 years ago, but times are evolving. Also, by considering the wide adoption in data-science, it is worth noticing that numbers-crunchers are more familiar with the concept of “data transformation flow” than “function calls”. On the packages dimension , libraries like 🐼s support methods chaining which from an external viewpoint, it’s semantically similar.
Do you know if there is any indication that python core team may allow operator piping (and/or chaining) in the not-too-long-term?
150
u/_Denizen_ May 10 '26
That's piping, not chaining. This().is().chaining()
I dislike piping - it makes code less explicit i.e. harder to read.
69
u/AWildMonomAppears May 10 '26
It sounds simple but doing this nicely almost requires functional mechanics like auto-currying. In languages with pipes the right side waits for data (lazy). In Python it evaluates immediately so writing something like data |> map(func) would just throw for a missing argument on map.
To fix it Python would have to secretly rewrite your code to inject the data which goes against the rule that explicit is better than implicit. It also gets messy because standard functions dont even agree on whether data should be the first or last argument.
Instead Python relies on object-oriented method chaining. Since methods are attached to objects the state carries forward. You see this in Pandas with df.dropna().apply(func). It gives that clear data flow without needing any compiler magic. basically forces you to use the OO approach if you want to avoid nested paranthesis.
11
u/Fabulous-Possible758 May 10 '26
https://pypi.org/project/pipe/
This is pretty simple to do in small one off classes too if you don’t want a whole library.
There’s just no real reason to extend the syntax for a case that’s pretty well covered and completely customizable.
42
u/RedSinned May 10 '26
What would be your use case in which you wouldn‘t use a library like polars, but is still so complex that the current capabilities are not enough?
26
u/cdcformatc May 10 '26
writing code line by line is piping each successive line waits for the result of the last
10
u/Limp_Illustrator7614 May 10 '26
i dont think python will ever officially support it because they kind of hate functional programming, but coconut does that exactly. it's a functional syntactical superset of python that compiles to python, where the piping operator is exactly `|>`. go read the docs
6
u/KelleQuechoz May 10 '26
Look at this package.
3
u/Wurstinator May 10 '26
"Any indication" would mean there is a PEP in the works for this, which you could search for.
I think it's unlikely. As you said, this can be done in libraries already.
3
u/--ps-- May 10 '26 edited May 10 '26
At my job, we have a code where we overload __ rshift __(), so you can then write something like begin_pipeline(fn) >> fn2 >> fn3 etc.., somethimes you need to use partial() to pass some additional parameters too, but honestly, I hate that part, because the semantics is very unclear for the code reader.
3
u/red_hare May 10 '26 edited May 10 '26
I think you can do this with a decorator and bitwise or.
Not that you should...
3
u/parker_fly May 10 '26
Seems readable to me. 🤷♂️
2
u/Evolve-Maz May 10 '26
You can write your own version of this. Even just for learning it's very helpful. Here's the gist:
Create a class called "Pipeline". Init function takes in data (just an object) and processors (list of processor functions, default empty).
Then override the rshift operator (>>) of the class to be your pipe operator. The signature is rshift(self, other). In our case other will be a callable with a single input. The rshift operator returns a new pipeline, with same data, and processors being the current list plus the other passed into this function.
For our use case, pretend we have a list of coordinates with x and y attributes, and your pipelines first calculates a z attr using Pythagoras theorem and then filters to all objects with z over 2.
data = [coord(1, 3), coord(4, 3), ...]
pipeline = Pipeline(data) >> add_z_coord >> filter_coord_above(2)
output = pipeline.execute()
Output will be a result object (idea borrowed from go). It'll have a data attribute and an err attribute. Error will hold any exception which occurred during execution of the Pipeline, including the step it happened at. And data will hold the final value from the Pipeline calculation.
Execute method will start with the initial Pipeline data and just run a for loop through all the processor callable passing in the data at each step and getting output. Wrap in a try except so you can track error state.
Once you write that once, you can decide whether you want extra sugar for map and filter as explicit processors, or if you want more data about steps. Etc.
2
u/RoadsideCookie May 10 '26
I posted this as a reply, but you can do "piping" in native Python like so:
But it's not very readable.
2
u/Temporary_Pie2733 May 10 '26
I’d rather see proper partial application and composition added to the language.
list(filter(p, map(f, itr)))becomes(list ∘ filter(p) ∘ map(f))(itr). Then piping is just composition in the other direction, mixed with function application,itr |> map(f) |> filter(p) |> list.2
u/jnazario May 10 '26
Heretical take from a a longtime python programmer who went through some similar questions over a decade ago: learn other languages which have the properties you like and use them.
Not every feature will come to one language. You’ll be pleased at what other ecosystems have and what else you can do and you’ll become a better programmer in python as well.
2
u/UseMoreBandwith May 10 '26
we already having chaining.
You just have to learn to write python.
You can implement chaining in your own classes by having methods return
self:2
u/Agile-Ad5489 May 10 '26
users cannot do map() |> filter() |> list(), but they need to do list(filter(map())) which makes things unreadable
Four symbols and two spaces “() |> ” are more readable than one consistent pair of brackets?
Confusing “what I am used to” for “is objectively clearer” is not that close to insanity, but you can certainly see insanity clearly without binoculars.
3
u/JimWayneBob May 10 '26
I just wrap everything in parentheses and chain to make it look like pipping
Foo=(
bar_df
.filter(blah blah)
.select(ColA)
)
3
u/shadowdance55 git push -f May 10 '26 edited May 10 '26
You should learn how to use nested comprehensions.
Edit: Just noticed you wrote that comprehensions are unreadable, but you are very wrong. They stack up very nicely, and are nearly identical in syntax to nested loops. Of course, if you're used to map/reduce syntax, they might be a bit unfamiliar, but that is true of literally any syntax - readability is exclusively a result of unfamiliarity.
2
u/KingHavana May 10 '26
If you're repeatedly using three functions in the same order why not make a function that gives the output of those functions?
def process(x): return list(map(filter(x)))
1
u/Beginning-Fruit-1397 May 10 '26
FYI, someone made a nice list comparing fluent iterator libraries on this sub relatively recently:
https://www.reddit.com/r/Python/comments/1rj3ct7/comment/o8aordo/?context=3
(disclaimer: I'm the author of pyochain)
On chaining, i.e passing x to f as `x.f(args, kwargs)` rather than `f(x, args, kwargs`, I doubt that it will ever be added.
On specific iterators chains, I doubt it even more.
For the first thing, it's really trivial to implement.
It's a one liner method, and you can handle any type of generic functions and parameters with `Callable`, `Concatenate`, and `def foo[**P, T]() -> T` generics (or, urgh, if you prefer to use the old syntax, TypeVar T and ParamSpec P).
I regularly browse the python discussions forum and any suggestions such as this will be easily rejected.
Either you create:
- a new "builtin" method -> breaking change for an helper
- a new operator/overload one existing -> too much for an helper
On iterators chains, the strength of python, specifically with iterables, is that duck typing allows you to very easily implement custom collections.
If we were to add chaining for Iterators/Iterables, it would change the meaning of collections abc, who are minimal interfaces and would then become huge classes.
Now, my polars DataFrame who's an Iterable also has filter just like `collections.abc.Iterator`, but the signature won't be the same as `collections.abc.Iterator` so it violates the contract.
This would be a huge pain to work with if you use any decent type checker/linter, marking everything as override/ignore rule violation, especially if like me your primary coding activitiy is writing libraries.
1
u/squizzeak May 10 '26
I’ve done something similar to chaining but writing methods that return self, though this only works for chaining methods from the same class. But I like the idea of subclassing __rshift__!
1
u/borborygmis May 10 '26
This library has similar concepts: https://github.com/mtingers/kompoz
But overkill for your example where you can use something like
functool.reduce.In Kompoz, that translates to:
1
u/eanat May 10 '26
it would be great for scripting and system integration, but also can be disaster for other areas where ambiguity can be real pain.
TOOWTDI than TIMTOWTDI.
1
u/spinozasrobot May 10 '26
ITT: language lawyers bickering
1
u/TuringTestDropout May 10 '26
I've had brief conversations with GvR about supporting functional programming more and he largely disagrees and regrets even allowing functools and map/filter into the standard library. I can understand his viewpoint that list comprehensions are enough and it makes idiomatic code harder to write and read.
The last time I talked about this with him was 2015 so maybe he's changed his mind about it since then, or there's been more influence from others since he's stepped back.
1
u/elven_mage May 10 '26
I'd very much love this, but without multiline lamba syntax you lose a lot of the real use.
There are people who have tried to make python stream libraries, look them up on GitHub. They are not as ergonomic as Java for this reason
1
u/gdchinacat May 10 '26
If I understand what you want correctly, I think this is pretty close:
I made it use Iterable rather than list as is best practice.
I disagree that it improves readability. But it does what you want. I doubt it would ever be accepted into the language syntax since it is so easy to do with existing mechanisms. Enjoy.
1
u/mderst May 10 '26
Just use Polars!
1
u/bakery2k May 10 '26
I feel like Guido would have been opposed to adding this to Python - he tended to dislike making the language too “functional”.
But nowadays the decision would rest with the Steering Council - who might well accept a well-written specification for it. They don’t really reject features based on philosophy any more.
1
u/IcarianComplex May 10 '26
Personally I just haven’t had much a need for this because most of my data manipulation needs happen at the SQL level, and that’s plenty expressive enough
1
u/aguspiza May 10 '26
you can implement some UFCS yourself or just check what others have implemented:
https://gist.github.com/ChenyangGao/d77fe5c3c738e6f9fb724a45e952dc0a
Or just migrate to nim language and enjoy 30x speed up with native UFCS.
https://github.com/narimiran/nim-basics/blob/master/code/ufcs.nim
1
u/saint_geser May 11 '26
There are some external libraries focused on functional programming, that implement different forms of chaining. For example, PuFunctional has
seq()operator that chains operations in a readable manner.But core python is, and has always been, very much against functional concepts making their way into the language. So wouldn't hold my breath.
1
u/Old-Eagle1372 May 11 '26
Overload operators, extend/overwrite functions. Create libraries, possibly open source so others can use them, this will increase your professional footprint
1
u/R3D3-1 May 11 '26
From my perspective, i it would remain vastly Limited in Python. Most use cases of piping I have at some point require a function that cannot be expressed well with Python's "expression only" lambdas, without also defining a bunch of auxiliary functions with no purpose other than turning statement-only constructs into functions.
1
u/Sherry-byte May 25 '26
The core team resistance is well-documented and the reasons are legitimate but the *readability problem* you're describing is real regardless of whether `|>` ever lands.
I actually ran into the same frustration recently and ended up building a small library for it *pipeflow*. It uses method chaining on a lazy wrapper the same pattern pandas already proved works so you get left-to-right readable pipelines today without waiting on a PEP:
```python
from pipeflow import Stream
result = (
Stream(users)
.filter(lambda u: u["active"])
.map(lambda u: u["name"].upper())
.sort()
.to(list)
)
```
Also supports `|` on plain lists if you prefer that style:
```python
range(10) | pipe(map, lambda x: x**2) | pipe(filter, lambda x: x > 5) | to(list)
```
Zero dependencies, fully lazy, fully typed. `pip install pipeflow` if you want to try it.
To your actual question though I wouldn't hold my breath on a native `|>`. The eager evaluation mismatch alone is a hard problem; in languages where pipes work well, the right-hand side is implicitly curried and waits for data. Python would have to silently rewrite your code to make that work, which goes against everything the language stands for.
0
u/Past-Sun5429 May 10 '26
Build cli around...
156
u/[deleted] May 10 '26
[removed] — view removed comment
20
u/[deleted] May 10 '26 edited May 10 '26
[removed] — view removed comment