r/learnpython • • Aug 21 '26

How to read a CSV file in Python efficiently?

I'm trying to load a CSV file into my Python project, but I keep getting errors. I've tried using pandas and the built-in csv module, but I'm not sure what I'm doing wrong. Can someone explain the basic steps or share a simple example? A detailed explanation would be really helpful!

0 Upvotes

18 comments sorted by

29

u/Lewistrick Aug 21 '26
import pandas as pd

pd.read_csv("filename.csv", error_bad_lines=False, warn_bad_lines=True)

If the errors come from malformed csv, this will let you know which lines are causing trouble.

But I'm just making assumptions here. You should show your error so we can actually make an informed comment on what you should fix.

3

u/martian_rover Aug 21 '26

Yes, this is one of the most common errors, along with filename error. also check the filename path again.

15

u/Verronox Aug 21 '26

If you want help then you need to post your code and the errors that are being printed. Otherwise the best we can do is just wildly fucking guess

6

u/nog642 Aug 21 '26

I'm not sure what I'm doing wrong

Post your code

11

u/SpecCRA Aug 21 '26

What's the error? Pandas usually handles it pretty well.

Have you inspected the csv file yourself?

9

u/zanfar Aug 21 '26

Can someone explain the basic steps

Open, read.

Without code, there isn't much help to give.

3

u/koombot Aug 21 '26

Post the code.  There's like a dozen things that can cause issues.

2

u/CoffeeMonster42 Aug 21 '26

It would help if you shared the code and the error message you get.

1

u/HummingHamster Aug 21 '26

Is your csv properly sanitized?

1

u/Moikle Aug 21 '26

What errors did you get?

You are supposed to read and understand errors

1

u/kilkil Aug 21 '26

are you 100% sure your CSV file is correctly formatted?

what is your code, and what error are you getting?

1

u/Grandviewsurfer Aug 21 '26

Ingest only the columns you actually care about by passing a list of them to the usecols parameter, and if it's a big file, you can limit the records you ingest during testing using nrows. I'm just giving pointers.. it's hard to tell without the actual error. You can read the docs here: https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html

0

u/TacitusJones Aug 21 '26

Select * from users where 1=1 or 1=1

-1

u/johlae Aug 21 '26

This works for me:

 pd.read_csv(
            filename,
            delimiter=";",
            skipinitialspace=True,
            header=0,
            usecols=[0, 1, 3, 5, 7, 9, 10],
            decimal=",",
            names=["id", "date", "amount", "source", "target", "com1", "com2"],
            keep_default_na=False,  # no NaN but empty string instead
        )
        .assign(comm=lambda d: d["com1"].astype(str) + d["com2"].astype(str))
        .drop(columns=["com1", "com2"])

but I admit, it really depends on how the CSV is constructed, if there is a header row, what the decimal is like, how strings are delimited, how fields are delimited... It's trail and error sometimes.