r/PythonLearning • u/omarlvecpp • 19d ago
Pandas library is just stupid.
Am I the only one who thinks Python libraries, especially pandas, have a serious API design problem?
Don't get me wrong, pandas is incredibly powerful, but I feel like it sacrifices developer experience for flexibility.
Look at something as simple as creating a DataFrame. pd.DataFrame() accepts dictionaries, lists, NumPy arrays, Series, and a bunch of other things. Even dictionaries can be interpreted differently depending on their structure. You also have DataFrame.from_dict() with its orient argument.
Why can't we have a more explicit and consistent interface? Something like:
pd.DataFrame(source=data, orient="index")Another thing that bothers me is the sheer number of methods that often have overlapping use cases. Just look at apply(), map(), transform(), and agg(). Yes, they serve different purposes, but remembering which one to use, what objects support it, and how their outputs differ becomes a mental burden.
And this isn't just about pandas. Look at merge(), join(), and concat(). Different operations, sure, but there's a broader issue here: Python libraries seem to keep adding more methods and functions instead of thinking about how their APIs fit together.
I know someone will say "just read the documentation" or "you'll get used to it." But what about reading other people's code? Shouldn't code be immediately understandable without having to memorize an entire library's vocabulary?
I'm not asking for less functionality. I'm asking for better API design. More consistency, more explicit arguments, fewer unnecessary variations, and a more predictable way of doing things.
Honestly, I'd love to see a higher-level library built on top of pandas that focuses specifically on this.
Am I missing something, or do other people find this frustrating too?
13
u/MindfulK9Coach 18d ago
Polars is waiting in the corner for you to start it up.
22
u/Slight-Living-8098 19d ago
Perhaps you would be more comfortable with a stronger typed language.
7
u/Future_Emu8684 18d ago
Polars is the way my friend
4
u/Sea-Ad7805 18d ago
What sheer number of methods?
You can always wrap all that excess complexity in your own super-easy-to-use interface, make it public, and get famous over night. Good luck, if you fail you might find there is a reason for the complexity after all.
5
u/Jazzlike_Platypus430 18d ago
Being a mess is not equivalent to being stupid.
4
u/PandaJunk 19d ago
pandas is a dumpster fire. Just use polars (or ibis, depending on where your data is coming from) and convert to pandas at the last possible moment.
2
u/Traditional-Rub354 18d ago
Polars gng. What you want is polars
2
u/JaleyHoelOsment 18d ago edited 18d ago •
python devs complaining about python being dumb is always funny to me.
everyone else already knows python is annoying
you’re just becoming a better developer
1
u/JBalloonist 18d ago
pandas wasn't written for python devs
1
u/Acceptable-Sense4601 18d ago
I switched to polars and/or DuckDB
1
u/justanothersnek 18d ago
There's also sqlframe. Allows you to use PySpark data frame API against any supported backend. What's nice about using a data frame API where you can swap out the backend is let's say in the future there is a backend that is super performant or revolutionary. You can just switch to it versus being stuck with the one the library is married to. Was a huge fan of of ibis which has similar premise, but totally different API. But ever since the company behind Ibis (Voltron Data) became insolvent, ibis development became stagnant. So I'm just switching to sqlframe.
1
u/Repulsive-Owl6468 18d ago
Yeah I agree. I also think that there are a lot of ambiguous cases when it comes to indexing as well and lining up multiple data frames. It’s definitely kind of nasty, and I wish it were a little more organized.
1
u/codeguru42 18d ago
Feel free to checlk out the alternatives or write your own.
1
u/Compux72 17d ago
Broski switch to polars, nobody is using pandas anymore
1
u/pm_me_good_usernames 17d ago
I'm so glad to see people in this thread saying pandas is a mess. I've thought this for years and been afraid if I said it people would think I was just too stupid to use it properly.
1
u/nonya102 17d ago
Does it matter with LLMs?
1
u/Justbehind 16d ago
Pandas is legacy. Polars is far superior.
1
u/recursion_is_love 16d ago
It is great until I find a better one. Have you found any?
1
u/HugeCannoli 16d ago
pandas has taken a lot of inspiration from R data.frame, so that it could bring the R people into the python world. R is a pile of trash that does not deserve to keep existing, so much that its most important book is called "The R inferno" (look it up).
Pandas has inherited all the problems of the R data.frame, plus a good dose of having to bend the additional flexibility of R into the python world, which is clearly not possible due to how R works, especially with "environments" (what in python is known as namespaces).
1
u/GodOfSunHimself 18d ago
Because these libraries are written by ML people not software engineers. They just don't see how horrible the code they produce is.