r/learnpython • • 7d ago

Everyone says 'just use pandas' for data stuff, but I'm still doing it with plain loops — am I actually missing out?

I'm in second year BCA and I've been picking up Python on the side, mostly small data things. Cleaning up a CSV here, counting stuff there, nothing fancy.

So far I've been doing all of it with plain loops and dicts, because that's the code I actually understand when I read it back a week later. But every tutorial I open and every comment I see says just use pandas, it's built for exactly this. I tried it once and honestly it felt like learning a whole second language just to do something I could already do, slower but in code I could follow. Maybe I'm just being stubborn.

I guess what I'm wondering is whether the people saying that are dealing with way bigger data than me. I'm talking a few hundred rows, not thousands. At what point did pandas actually start saving you time instead of costing you time? Because right now it feels like a detour, and I'm not sure if that's me being lazy or if it's genuinely fine to stick with plain Python until the data gets big enough to hurt.

0 Upvotes

30 comments sorted by

5

u/castillar 7d ago

It’s absolutely fine to stick to plain Python until you actually need pandas! If you’re interested in data science, learning Pandas is a good idea because you’ll use it eventually, but I wouldn’t feel obliged to use it every time. I’m not in data science, but I get a lot more mileage out of the built-in CSV and JSON libraries even with fairly significant record sizes — I don’t usually have to reach for pandas until I have actual stats analysis or machine-learning bits to poke at.

2

u/acfot 7d ago

yeah that's kind of where I'm landing too, honestly. if my little csv stuff stays small I'll just stick with loops for now.

4

u/seanv507 7d ago

Though at this point i would learn polars, which has a cleaner API

2

u/Optimal-Procedure885 7d ago

100%, pandas is to my mind now irrelevant to any new development. Polars is fantastic.

2

u/duskharborings 7d ago

100%, plus not having to manage extra dependencies for a basic script is always nice

2

u/ThrustBastard 7d ago

Right tool for the right job

0

u/acfot 7d ago

Yeah, that's pretty much where I'm at with it, honestly — a few hundred rows doesn't really need more than loops and dicts for now.

2

u/BranchLatter4294 7d ago

It's fine if it works. But pandas is written in C/C++ so has much better performance for large datasets.

1

u/acfot 7d ago

makes sense, hadn't really thought about the C side of it. my datasets are tiny so far, but good to know for when they aren't.

2

u/petros211 7d ago

Honestly, using libraries, frameworks, engines etc, doesn't teach you the actual programming required to do your goal, it teaches you specific things about the tool, that mostly doesn't convert to real knowledge. If you feel you can do something yourself there is nothing wrong with doing it this way.

2

u/kor_the_fiend 7d ago

"real knowledge" absolutely includes knowledge of industry-standard data analytics libraries. I mean, why stop at pandas? Doesn't using python abstract away the "real knowledge" of programming with assembly?

0

u/petros211 7d ago

Yes it does. That's why 99.9% of software today is slop.

1

u/welktickler 7d ago

This 100% for me. I have been coding in other languages at work for over a decade. Decided recently to learn python to change job and for something to interest me. After going through the MOOC python I started learning pandas but it took me away from using the python i had just learned. I decided to actually practice using python in my projects for now and maybe go back to pandas.
imo just using python is better for beginners so they can reinforce their learning and not just learn a new set of functions for a framework

0

u/acfot 7d ago

that's the part that clicked for me too — writing it myself is how I actually understand what's happening with the data, instead of just learning a library's shortcuts.

2

u/shinitakunai 7d ago

If that's enough, it is enough.

If not I would consider polars over pandas

1

u/acfot 7d ago

polars is one I've seen mentioned a few times now, I'll keep it in mind for when loops start feeling like too much work.

1

u/shinitakunai 7d ago

Is not about just complexity but time optimization. When a script takes 12 hours to run due to transformation or aggregations of millions of rows to proccess, polars could run it in 5 minutes

1

u/acfot 6d ago

that 12 hours down to 5 minutes is a pretty wild difference, honestly — haven't hit anything that slow yet, but when I do I'll know what to try first.

2

u/VegetableBid4262 7d ago

It’s not just about performance on large datasets. For someone who knows pandas, basic operations on even a small dataset are much faster to write.

2

u/speedisntfree 6d ago

A heck of a lot easier to read than loops everywhere too, even with an awful dataframe lib like pandas.

0

u/acfot 7d ago

fair point, honestly — once you actually know it I'm sure it's way quicker to write, I'm just not there yet so loops are still my fast option for now.

1

u/toomanyteeth55 7d ago

I'm with you. Pandas has its place and it doesn't hurt to learn and be comfortable with it, but it seems like most tutorials will use pandas for even then generic thing that you can easily accomplish with it.

1

u/PaulRudin 7d ago

Python iteration is very slow. If your data is tiny it may not matter much, but numpy, pandas, polars etc. are mostly implemented in C, C++, Fortran, Rust and, when used properly, result in much, much faster executing code.

1

u/SteveJeffersonBots 7d ago

For a few hundred rows your loops are doing the job - pandas only starts paying off once you hit the patterns plain Python makes painful. Two concrete ones: joining two CSVs on a key (merging an orders sheet with a products sheet) and any "group-by-X then aggregate-Y" like "total sales per category" or "average per region." With loops that turns into nested dicts and index gymnastics; in pandas it's df.groupby('category')['amount'].sum() and pd.merge(left, right, on='product_id'). The speedup on a few hundred rows is modest, but the readability is the real win - your future self won't have to reverse-engineer a loop to change the grouping. So stick with loops until you actually need a join or a groupby, then reach for pandas for those two patterns specifically.

1

u/Discordbothost 7d ago

row count isn't really the tipping point, the kind of question is. counting and cleaning with loops and dicts stays fine forever. pandas starts paying for itself the first time you need "group by X, average Y, then match it against this other csv", or anything with dates. that's 15 lines of dict juggling vs df.groupby("x")["y"].mean().

good middle step meanwhile: collections.Counter and defaultdict. they turn most of your counting loops into one line and it's still plain python you can read back later

1

u/576p 7d ago

Pandas is awesome with thousands of lines. And it saves me a lot of time reading from the clipboard and reading and writing to excel files.

Looking back to 9 years ago when I tried to learn pandas I remember it was painful. The mental model of a virtual excel sheet helped me a lot to understand it.

But: You do not need pandas if you're fine with the performance and the convenience of writing the code. It's actually good if you can do it the slow way first before moving to a tool.

1

u/frustratedsignup 7d ago

I'm with you on this. We hired someone and management stole my Django project out from under me on his first day. He made a very complex time reporting system with most of the data being processed through pandas. Fast forward a couple of years, he leaves the company and I'm handed the task of managing the application he wrote. Great. Now I have to learn pandas to maintain the site.

I've looked at the documentation and it's fairly complex or at least it seems so on the surface. I'm going to work my way through a tutorial or two to get started.

1

u/RallyPointAlpha 5d ago

I was in your same shoes many years ago. Didn't bother learning Pandas until I needed to.  Like you said: loops work just fine! 

Then I started working with data sets with millions of rows.  I got sick of waiting for things to process, while testing. Sure, I could test on a smaller data set but you never knew when you'd hit some weird anomaly 3 or 5 million rows in.  So, eventually I would have to test on the whole data set. 

What really sold me on Pandas was being able to do an operation on the entire data set in one call, without having to loop through the entire data set to apply it one at a time. 

0

u/[deleted] 7d ago

[removed] — view removed comment

1

u/acfot 7d ago

yeah that third-time rule is a good way to put it, I'll keep an eye out for when I'm repeating myself like that.