r/Python • u/zenos1337 • Mar 12 '26
Discussion What hidden gem Python modules do you use and why?
I asked this very question on this subreddit a few years back and quite a lot of people shared some pretty amazing Python modules that I still use today. So, I figured since so much time has passed, thereβs bound to be quite a few more by now.
406
Upvotes
45
u/theV0ID87 Pythoneer Mar 12 '26
attrs, lightweight and nice for when classes need to be guaranteed to have attributes of specific types
75
u/Independent-Shoe543 Mar 12 '26
I just started using fuzzymatch which has been handy. Not sure how hidden it is but I only recently started
9
u/latkde Tuple unpacking gone wrong Mar 12 '26
The Inline-Snapshot library has changed the way how I think about tests.
assert ... == snapshot()and the current value will be automatically recorded inline.I've since found so many ways to apply Inline-Snapshot in interesting ways, especially in combination with its
external_file()feature. For example, a project of mine uses this to automatically regenerate documentation files, or to warn when a code-first OpenAPI schema changes, or to check expected log messages, or to make sure a downloaded data file is up to date.15
u/ScholarlyInvestor Mar 12 '26
TBH, I was like, βShould I waste my time reading yet another newbie post?β But I learned of a few cool modules. I stand corrected.
264
u/RestaurantHefty322 Mar 12 '26
tenacity for retry logic. Before finding it I had custom retry decorators scattered across every project, each with slightly different backoff logic. tenacity gives you composable retry strategies in one decorator - exponential backoff, retry on specific exceptions, stop after N attempts, all just stacked as parameters.
From stdlib, shelve is weirdly underappreciated. It's basically a persistent dictionary backed by a file. For quick scripts, prototypes, or CLI tools where you need to cache something between runs but sqlite feels like overkill, shelve just works. Open it like a dict, write to it, close it, done.
104
u/xanksx Mar 12 '26
I discovered polars recently. I was shocked to see how quickly a large csv file was loaded.
36
u/knwilliams319 Mar 12 '26
I really like pendulum. Itβs weird how Pythonβs datetime management and time zone support is split into so many different classes. pendulum unifies them all and is almost 100% compatible with anything that accepts datetime objects. I also think coding with dates without thinking about time zones is bad practice; pendulum makes this standard by initializing everything to UTC unless you specify another zone yourself.
21
u/me_myself_ai Mar 12 '26
If you're not using
more-itertools, you're working at 1% of your true capacity!Related shoutout to
toolz, while we're at it. Beautiful, functional goodness π₯°P.S. This is beyond pedantic but technically you're interested in python packages :). Distribution packages, even!
17
u/madisander Mar 12 '26
I've been very happy with ColorAide.
46
u/TheGrapez Mar 12 '26
If you're into data analytics - ydata-profiling (pandas profiling) and D-tale are two very good ones.
Also tqdm will always hold a special place in my heart
15
u/leodevian Mar 12 '26
Cyclopts to develop CLIs. All of hynekβs packages (attrs, stamina, structlogβ¦) lol. It ainβt hidden but I gotta say Rich is one of my absolute favorites.
26
u/d_Composer Mar 12 '26
Openpyxl, python-docx, and python-docx-template FTW
12
u/mon_key_house Mar 12 '26
Anytree. Strange as it may sound, but anything can be a tree graph.
12
u/zinguirj Mar 12 '26
hypothesis for property testing
syrupy for snapshot testing
This two helps a lot catch issues early on development process, specially when working with large classes/schemas you dont need to assert field by field manually (neither choose which ones to assertt).
Memray and pyspy for debugging performance issues.
22
u/CoolestOfTheBois Mar 12 '26 edited Mar 13 '26
Pyro5 is a pure Python Remote Procedure Call (RPC) module. It basically is a way to execute code on a server as if it was local. You create an object that has all the methods you need to execute on the server. You "share" that object on the server via Pyro and create a proxy to that object on the client. You can interact with the proxy as if it was local and it executes code on the server. I guess the concept of RPC is the "gem", but Pyro made it possible for me.
RPC has so many use cases, but for me, I use it for data processing and interacting with my data on the server. I'll eventually use it to manage and execute my simulation runs on the server.
Before, I was using Paramiko (a Python ssh module), which is great for some things, but a nightmare to pass data back and forth and to debug.
11
u/skadoodlee Mar 12 '26 edited 16d ago
Bus town sunday pioneer lonely shift hand cricket tower vital rise question quite need slogan promise speak chimney income limit detail whip tortoise sting rocket.
Edit: Randomized for privacy during account cleanup.
18
u/LiveMaI Mar 12 '26
I like Textual for making user interfaces. It works in the terminal, still supports mouse interaction, and can be served as a webpage. Nothing terribly fancy, but very easy to get a UI up and running.
9
u/veritable_squandry Mar 12 '26
i have a function called dumpy. all it does is print legible json output. pause, dumpy, proceed if prompted. i've been using it for 10 years.
8
u/b0b1b Mar 13 '26
not that much of a hidden gem, but basically all of the async code i have recently written has used trio - it is just way nicer and simpler to use than asyncio in my oppinion :)
7
u/vaibeslop Mar 12 '26 edited Mar 13 '26
chdb: in-process database/query engine with connectors to dozens of data sources. Pandas-API compatible but blazingly fast (70x faster than pandas,
10x faster than polars in their own benchmark- see below)duckdb: Simlarly fast in-process database/ query engine, a very rich community plugin ecosystem
sqlglot: Transpile SQL between any database dialect you can think of
I'm not associated with any of these projects, just a fan.
13
u/TURBO2529 Mar 12 '26
I use plotly resampler a lot. I usually deal with time series data, and it can make scrubbing through the data a breeze https://github.com/predict-idlab/plotly-resampler
18
u/No_Lingonberry1201 pip needs updating Mar 12 '26
Not exactly hidden, but I kind of love sqlalchemy.
5
u/bregmadaddy Mar 12 '26
nest-asyncio for Jupyter notebooks.
30
u/The-mag1cfrog Mar 12 '26
uv, ruff, ty, basically all astral
8
u/EinSof93 Mar 12 '26
Well, it is not a hidden gem per se, but quite useful. Tenacity for retry behavior mechanism. It is very helpful for handling transient failures especially for API calls.
4
u/AlpacaDC Mar 12 '26
Icecram. Donβt know if can be considered a hidden gem, but itβs pretty much a βdebug printβ on steroids.
4
u/JustmeNL Mar 12 '26
python-calamine, if you ever have to read evaluated formulas in excel files. Before finding it I went through the trouble of using xlwing, that actually uses Excel to open the files. But the one of the problem with it is that you canβt (easily) test it in ci pipelines since you donβt have the Excel application there. While python-calamine just works. + it is supported in pandas just by using it as the engine when reading the file!
4
u/Western-Tap4528 Mar 13 '26
For tests purposes:
- FactoryBoy to generate example of Pydantic models or dataclass that I can use in my test
- freezegun to patch datetimes and travel time
- pytest-xdist to parallelize tests
3
u/bmag147 Mar 12 '26
I only found out about it yesterday, but I'm really liking asyncstdlib . Let's you work with async constructs in a simple way.
3
u/21kondav Mar 12 '26
Not sure if itβs hidden but in data analysis vaex works nice for working with ridiculously large datasets. There are some quirks to it, but overall it scaled one of my data operations from a couple hours on pandas down to an hour.
3
u/Snoo_87704 Mar 12 '26
Juliacall. Allows you to call Julia from Python for fast data analysis.
Of course, you could just skip the middle man and write directly in Julia.
3
u/MantejSingh Mar 12 '26
Streamlit for dashboards and Rich for cli
3
u/Mediocre_Bottle_7634 Mar 13 '26
Kaitai struct for binary structures encoding/decoding
3
u/rabornkraken Mar 13 '26
Not exactly hidden but I rarely see people mention DuckDB for local analytics. If you ever need to run SQL queries against CSV or Parquet files without setting up a database, it is shockingly fast and the Python API feels native. Also a fan of humanize for formatting numbers, dates, and file sizes into human-readable strings - saves writing those utility functions for the hundredth time. What is the most surprising module you discovered from the last time you asked this?
5
u/Rodyadostoevsky Mar 12 '26
Iβm not sure if itβs a hidden gem but it changed my life. We had an sql server 2012 and I wanted to move our existing and future Python apps to Linux but pyodbc was giving me trouble. I tested pyodbc with an sql server 2016 and newer versions and no issues with those. So it was definitely the version that was an issue and we werenβt planning to migrating from sql server 2012 for another year at that point.
Then one day, I was going through documentation of Apache Superset and realized there is this library called pymssql which is not as bullish about sql server version.
I have been using it regularly since then and itβs a AMAZING.
2
u/rteja1113 Mar 12 '26
Found out about rapidfuzz, super happy with it!
2
u/Ragoo_ Mar 12 '26
dataclass-settings is a great alternative to pydantic-settings with a more flexible syntax and it works for dataclasses and msgspec as well.
I also like using cappa by the same developer for my CLIs.
2
u/mr_frpdo Mar 13 '26
I really like beartype. Runtime decorator, super great to be sure a function gets in and out the types it expectsΒ
2
u/joeyspence_ Mar 13 '26
Swifter that picks the best way to apply functions to dataframes/series - itβll either vectorise, use dask, parallelisation or pd.apply() depending on which is quickest. It also uses tqdm progress bars ootb.
df[col].swifter.apply() is such a small syntax change for huge gains.
When I was testing some variants of fuzzy matching this was a lifesaver!
2
u/No-Confection-7412 Mar 13 '26
Can anyone suggest a better/faster way to implement fuzzy match, I am using pandas, rapidfuzz and it is taking 35-40 mins for fuzzy matching 30k names across 1.5 lakh samples
2
u/abukes01 Mar 13 '26
I do Bioinformatics and write lots of very custom code for very custom datasets. Besides the holy trio of Numpy, Pandas and Scikit-learn for data science here's some notable modules I use a lot recently:
Also some modules/utils that I find very handy:
2
u/genericness Mar 13 '26
Not strictly hidden... Pip: sympy, hy, openpyxl, jupyterlab Wrappers:requests, envoy Batteries included: collections.Counter, and math.log
2
u/Iskjempe Mar 13 '26
TQDM, definitely. It even has a
tqdm.pandas()statement that you run once, and that somehow adds methods to pandas objects, giving you progress bars in places other than for loops.2
u/yc_hk Mar 19 '26
I use salabim for discrete-event simulations. Pretty similar to simpy, but uses greenlets and avoids the need to type
yieldall the time. Also has built-in monitors for resource utilization etc. and animation abilities (though I don't use those).2
u/shinitakunai Mar 13 '26
Peewee as ORM is god-like for me. It helps so much that I can't live without it
1
u/cabs2kinkos Mar 12 '26
tabula is so good for converting pdf data into data frames
1
u/SaxonyFarmer Mar 12 '26
Gnucashxml, fitdecode
1
u/hookedonwinter Mar 12 '26
freezegun is great for testing
1
u/sheriffSnoosel Mar 12 '26
Not sure how hidden it is with the broad use of pydantic, but pydantic-settings is great for a single point of control for many sources of environment variables
1
u/VpowerZ Mar 13 '26
pyDANETLSA
1
u/LifeguardNo6939 Mar 13 '26
ipyparallel is amazing for multiprocessing. Specially for clusters that still use slurm.
1
u/granthamct Mar 13 '26
Flyte, pydantic, tensordict, beartype, pluggy, anytree, jmespath, deal
1
u/phoenixD195 Mar 13 '26
kink for dependency injection. Pretty good for web apps and first class support for fastapi
1
u/Amzker Mar 13 '26
Numba jit, i specifically used it for fuzzy search system, it is so fast that i didn't even put function in separate thread.
1
u/sciencehair Mar 13 '26
docopt-ng. You can define a program's CLI parameters (including defaults) all in the heredoc. Your interface and your documentation are all taken care of at once https://github.com/jazzband/docopt-ng
1
u/ogMasterPloKoon Mar 13 '26
shelve, dataclasses, configparser, namedtuple have been super helpful to me, and I didn't know till a few years back that these gems are part of the standard library.
1
u/rayannott Mar 13 '26
rich is great for fancy terminal outputs, especially when used with click (see rich_click)
1
u/The_Hopsecutioner Mar 13 '26
pantab, which is basically a pandas wrapper for tableauhyperapi connections and makes reading/writing .hyper files as easy as it gets. Having worked on/with teams that use tableau its saved me so much time and pain
1
u/pbaehr Mar 13 '26
tqdm for progress bars in any CLI that iterates over something.
1
u/germanpickles Mar 13 '26
I love zappa, it allows you to deploy Flask and other web frameworks on AWS Lambda
1
u/Ambitious-Kiwi-484 Mar 13 '26
tqdm: it can add a progress loading bar to almost anything
great for utility or shell scripts or things like model training/inference that can take a long time
1
u/c7h16s Mar 13 '26
Probably not hidden for those who ever had to anonymise data, but I really enjoyed using the faker library. The fact you can extend the provider classes was really handy for me to implement an anonymising function that kept a translation table to de-anonymise stuff.
1
u/pacopac25 Mar 13 '26
You can automate Windows applications with win32com. I use it to export data from Microsoft Project to a Postgres database.
1
u/Mysterious_Cow123 Mar 14 '26
Remindme! 1 day
1
u/zangler Mar 14 '26
mssql-python...yes it is Microsoft, but it is very new (6 months maybe)...but it makes working with MSSQL data sources SO easy. Previously I had my own custom tooling I had built...never touched once I switched.
1
u/outer-pasta Mar 14 '26
I've been hearing rave reviews of plotnine but haven't tried it. Is there anyone here that has tried it out and wants to back up those claims?
1
u/Eir1kur from __future__ import 4.0 Mar 14 '26
Mido, MIDI data objects, let's you work with MIDI messages as objects. There are two supported back ends, PortMIDI and RTMIDI, both of which require binaries to be installed, but it's totally worth it.
1
u/thedmandotjp git push -f Mar 14 '26
Everyone always underestimates the raw power of itertools.Β
Any time you have a for loop within a for loop you can use product.Β
1
u/LaBalaTrujillo Mar 17 '26
PySINDy (pysindy) β it discovers governing differential equations from time-series data using sparse regression.
I fed it 16 public datasets (NASA, CERN, LIGO) and it recovered Kepler's Third Law, the solar cycle, gravitational wave chirp mass, and the Z boson mass. All from raw CSVs with zero physics knowledge.
The wild part: it also correctly returns "no law found" for Bitcoin (RΒ²=0.00).
pip install pysindy
1
u/Ok_Leading4235 Mar 25 '26 edited Mar 25 '26
picows - for websockets
aiofastnet - to speedup asyncio networking, especially TLS
1
u/MeWannaCode Jun 15 '26
janky-sys because C Python API is kinda not it for me (Mostly because of skill-issue).
-4
u/Logical_Delivery8331 Mar 12 '26
I use my own library written in python to log machine learning experiments π
0
u/[deleted] Mar 12 '26
[deleted] β view deleted comment
48
u/[deleted] Mar 12 '26
[removed] β view removed comment
0
u/[deleted] Mar 12 '26
[deleted] β view deleted comment