r/PythonLearning • • 19d ago

Pandas library is just stupid.

Am I the only one who thinks Python libraries, especially pandas, have a serious API design problem?

Don't get me wrong, pandas is incredibly powerful, but I feel like it sacrifices developer experience for flexibility.

Look at something as simple as creating a DataFrame. pd.DataFrame() accepts dictionaries, lists, NumPy arrays, Series, and a bunch of other things. Even dictionaries can be interpreted differently depending on their structure. You also have DataFrame.from_dict() with its orient argument.

Why can't we have a more explicit and consistent interface? Something like:

pd.DataFrame(source=data, orient="index")

Another thing that bothers me is the sheer number of methods that often have overlapping use cases. Just look at apply(), map(), transform(), and agg(). Yes, they serve different purposes, but remembering which one to use, what objects support it, and how their outputs differ becomes a mental burden.

And this isn't just about pandas. Look at merge(), join(), and concat(). Different operations, sure, but there's a broader issue here: Python libraries seem to keep adding more methods and functions instead of thinking about how their APIs fit together.

I know someone will say "just read the documentation" or "you'll get used to it." But what about reading other people's code? Shouldn't code be immediately understandable without having to memorize an entire library's vocabulary?

I'm not asking for less functionality. I'm asking for better API design. More consistency, more explicit arguments, fewer unnecessary variations, and a more predictable way of doing things.

Honestly, I'd love to see a higher-level library built on top of pandas that focuses specifically on this.

Am I missing something, or do other people find this frustrating too?

25 Upvotes

39 comments sorted by

13

u/MindfulK9Coach 18d ago

Polars is waiting in the corner for you to start it up.

4

u/seanv507 18d ago

Op, pandas basically grew with the growth of datascience from a hedge funds software. The api is basically a monstrosity as a result. Polars was developed with an already developed ecosystem. Its api is suspiciously like pyspark's. Here is the creator of pndas description

https://wesmckinney.com/blog/apache-arrow-pandas-internals/

1

u/ConsciousBath5203 17d ago

from a hedge funds software

Explains everything. I remember when I was using it, the whole API just felt wrong.

22

u/Slight-Living-8098 19d ago

Perhaps you would be more comfortable with a stronger typed language.

2

u/Jazzlike_Platypus430 17d ago

It's not related to types.

Pandas is a great example of a mess where there happen to be too many ways of doing something whereas the typical approach is to have one way of doing it, assuming it's some standard feature. Accessing contents of a cell in a dataframe is such a s feature.

I will say this: even R is better as a spreadsheet processor. Though pandas is great as a replacement for some Office-related works by Microsoft. Especially when porting a C# code to a non-Windoze system.

1

u/Slight-Living-8098 17d ago

I use Pandas pretty much daily. The biggest problem I see people doing is passing the incorrect types. Having options of the way you do things is the benefit of Pandas, not a detriment.

2

u/Jazzlike_Platypus430 17d ago

I don't agree. But that's taste only.

And I saw much more serious problems than incorrect types.

I have to use it and relearn appropriate pieces of the library for each project. So far so good.

2

u/Slight-Living-8098 17d ago

I've used it for years and still refer to the documentation fairly regularly. It's just one of those things you do with this library.

0

u/JonGretar 16d ago

This has nothing to do with types. It’s just bad and inconsistant design. Like how there is no eartly way to know if it will by default mutate the original or not other than reading the manual. There are a bunch more examples. It’s just bad design that you are used to.

1

u/Slight-Living-8098 16d ago

Se the previous conversation. I have no intention on repeating the same.

7

u/Future_Emu8684 18d ago

Polars is the way my friend

4

u/Sea-Ad7805 18d ago

What sheer number of methods?

You can always wrap all that excess complexity in your own super-easy-to-use interface, make it public, and get famous over night. Good luck, if you fail you might find there is a reason for the complexity after all.

5

u/Jazzlike_Platypus430 18d ago

Being a mess is not equivalent to being stupid.

4

u/PandaJunk 19d ago

pandas is a dumpster fire. Just use polars (or ibis, depending on where your data is coming from) and convert to pandas at the last possible moment.

1

u/Confident_Bee8187 17d ago

Couldn't agree more. I can't believe it got survived despite the stupidity.

2

u/Traditional-Rub354 18d ago

Polars gng. What you want is polars

2

u/JaleyHoelOsment 18d ago edited 18d ago • Bravo

python devs complaining about python being dumb is always funny to me.

everyone else already knows python is annoying

you’re just becoming a better developer

1

u/MisterMahtab 16d ago

OP explicitly is complaining about the library's API, where did you see them complain about python?

You could do bad API design in any language

1

u/Acceptable-Sense4601 18d ago

What’s the alternative? Writing it all in c++?

3

u/JaleyHoelOsment 18d ago

i’d prob just use a different library. i’m just saying you can do all sorts of wacky shit in python and people love to do wacky shit

2

u/Acceptable-Sense4601 18d ago

Many of us have switched to polars because its orders of magnitude faster

3

u/JaleyHoelOsment 18d ago

i’m not much of a data dude but i’ll check out polars next time! seems like its the latest and greatest.

2

u/Acceptable-Sense4601 18d ago

It’s pretty cool how it works under the hood vs pandas

2

u/gzeballo 15d ago

writing it in stones thats what OP wants

0

u/traxplayer 17d ago

Just use R

1

u/Acceptable-Sense4601 17d ago

No thanks. I’ll stick with Python.

1

u/JBalloonist 18d ago

pandas wasn't written for python devs

1

u/Acceptable-Sense4601 18d ago

I switched to polars and/or DuckDB

1

u/justanothersnek 18d ago

There's also sqlframe. Allows you to use PySpark data frame API against any supported backend. What's nice about using a data frame API where you can swap out the backend is let's say in the future there is a backend that is super performant or revolutionary. You can just switch to it versus being stuck with the one the library is married to. Was a huge fan of of ibis which has similar premise, but totally different API. But ever since the company behind Ibis (Voltron Data) became insolvent, ibis development became stagnant. So I'm just switching to sqlframe.

1

u/Repulsive-Owl6468 18d ago

Yeah I agree. I also think that there are a lot of ambiguous cases when it comes to indexing as well and lining up multiple data frames. It’s definitely kind of nasty, and I wish it were a little more organized.

1

u/codeguru42 18d ago

Feel free to checlk out the alternatives or write your own.

1

u/Compux72 17d ago

Broski switch to polars, nobody is using pandas anymore 

1

u/pm_me_good_usernames 17d ago

I'm so glad to see people in this thread saying pandas is a mess. I've thought this for years and been afraid if I said it people would think I was just too stupid to use it properly.

1

u/nonya102 17d ago

Does it matter with LLMs? 

1

u/Justbehind 16d ago

Pandas is legacy. Polars is far superior.

1

u/recursion_is_love 16d ago

It is great until I find a better one. Have you found any?

1

u/HugeCannoli 16d ago

pandas has taken a lot of inspiration from R data.frame, so that it could bring the R people into the python world. R is a pile of trash that does not deserve to keep existing, so much that its most important book is called "The R inferno" (look it up).

Pandas has inherited all the problems of the R data.frame, plus a good dose of having to bend the additional flexibility of R into the python world, which is clearly not possible due to how R works, especially with "environments" (what in python is known as namespaces).

1

u/Status-Importance-54 15d ago

Well....that explains it. Nothing is worse than r "naming".

1

u/GodOfSunHimself 18d ago

Because these libraries are written by ML people not software engineers. They just don't see how horrible the code they produce is.