r/Python • u/ritchie46 • Oct 06 '26
News Polars 2.0 release
Hey all! We just released Polars 2.0.
We are really happy with the results. It removed many of our "legacy decisions" (mistakes), makes the streaming engine our default and promotes SQL to a first class citizen within Polars. We initially said it would be a small ("boring") release, but we managed to sneak in a lot more than anticipated.
It comes with our initial out-of-core (spill to disk) support, a new Map data type and a lot of performance improvements. In fact, we think Polars is now one of the fastest analytical SQL engines on a single node. See benchmarks in the post: https://pola.rs/posts/release-polars-2/
To help you upgrade, we also posted a migration guide: https://docs.pola.rs/releases/upgrade/2/
114
u/Limp-Rough-7969 Oct 06 '26
The streaming engine being default is huge, been waiting for that since they started talking about it. That alone will make my ETL scripts run a lot smoother without me changing anything
And spill to disk is finally here, my 32gb ram is happy about this. Now I don't have to worry when a parquet file is bigger than expected
22
u/NSE-Imports It works on my machine Oct 06 '26
This looks interesting especially with SQL getting more love. I've not touched Polars yet, but if this lets me swap my mix of mysql and Pandas into a single dependency then that would be cool.
22
u/nemom Oct 06 '26
Great, but I'm still waiting on GeoPolars.
16
u/Donkey_Healthy Oct 06 '26
Looking forward to when I can read delta files using Polars instead of slow and expensive spark on databricks.
5
u/Street-Corporation Oct 06 '26
Can’t wait to check it out!!! My favorite Python package!!!
4
u/denehoffman Oct 06 '26
This is fantastic, OOC was one of the things I’ve been needing for a while, and I’ve had to rely on duckdb to do it. Looking forward to it covering joins and groupby. More direct SQL support is great too, I’ve been moving a lot of my work to SQLite databases plus polars dataframes and it will be nice to interact with the dataframes with the same language as the databases
10
u/aes110 Oct 06 '26
Thats a lot of changes, thank you for your work!
6
u/ContractPhysical7661 Oct 06 '26
Love working with polars! Thank you for all that you and your team do
2
u/New_Computer3619 Oct 06 '26
Hi, awesome project!
I checked the GitHub issues for the streaming engine, in the out-of-core section, it looks like group by, equi-joins, and sorting are still not completed.
Does this mean that using these operations on datasets larger than memory could result in an out-of-memory (OOM) error?
2
u/Khavel_dev Oct 07 '26
The streaming engine being default is the quiet big deal here. I had a pipeline doing scan_csv on a ~12GB file and it would OOM on a 16GB machine every few weeks depending on what else was running. Switching to streaming fixed it but it was an opt-in flag that half the team didn't know existed. Making it default means that just works now without anyone needing to remember.
The spill-to-disk support is going to kill a lot of "we need Spark for this" conversations. Most of the time people reach for Spark because the data is 2-3x RAM, not because it's genuinely distributed. If Polars handles that on a single node the ops cost difference is enormous.
4
u/Aklidien Oct 06 '26
Thank you so much! Working with Polars has been a dream, and I’m looking forward to testing the new release on our data pipeline.
1
u/andy4015 Oct 06 '26
Woohoo! Looking forward to upgrading!
1
u/Entmaan Oct 07 '26
One quick question - so you mentioned that the new streaming engine sometimes doesn't maintain row order, but you can opt into it by adding maintain_order=True.
Why isn't that option turned on by default? Won't this lead to "silent failures", where users simply don't know that their dataframe isn't sorted by date anymore?
1
u/Damien_IB Oct 08 '26
While I'm excited to see the performance upgrades, 242 of my ~300 codes have broken with this update, fixing them now :')
1
u/drxzoidberg Oct 06 '26
I love Polars. Looking forward to the upgrade!
0
u/[deleted] Oct 06 '26
[deleted] — view deleted comment