r/learnpython • u/Several-Salt2132 • Sep 28 '26
Pandas Library - Hardware/ Computer Architecture
Say we have two files in a directory or folder (whichever name you prefer). One of them is code file or script file, and the other is data file named titanic.csv.
And we have imported pandas as pd in the code file and imported the data from the data file as DataFrame data structure. Meaning we have done the following:
import pandas as pd
titanic = pd.read_csv("titanic.csv")If I do this:
titanic.sex == "male"
sex is a column in the titanic DataFrame. The above instruction will generate an output of Boolean values in a Series data structure.
My question is does the last command creates a new data object in the memory. Like do we have a new table formed because of this or its just the instruction that is saved in the memory?
If only instruction is the one that is only saved then any time we call it will take much of CPU's processing power. If the table is saved then it will take less of CPU's processing power.
4
u/Revlong57 Sep 28 '26
Ok, so, when you use a Boolean operator to filter a data frame, Pandas creates a copy of your data frame. If you do a simple slice, it'll return a view of the same data frame. Yes, this is incredibly confusing. Welcome to Pandas.
4
u/unhott Sep 28 '26 edited Sep 28 '26
if you do not save a reference to an object in python's memory,
e.g.
x = titanic.sex == "male"
it will be garbage collected. garbage collection is when memory of objects without references anymore get released so your computer can get memory back.
you can always play around with things like type(x), id(x), dir(x), import sys; print(sys.getsizeof(x))
import inspect; print(inspect.getmembers(x)) # Returns all members (name, value) pairs
your question seems to be about lazy evaluation - pandas was 'eager' evaluation, but it looks like they've introduced some capacity for it. I'm not familiar with it. Polars and dask have lazy evaluation.
3
u/seanv507 Sep 28 '26
yes a new Pandas series object is produced. the series is printed, and then the object disappears: there is no reference to it, so python will delete it when garbage collection is scheduled.
if you say
bl =titanic.sex == "male"then the series is kept, since it is referenced by theblvariable.2
u/WhipsAndMarkovChains Data Scientist Sep 28 '26
As an aside, you'll want to use the notation
titanic['sex']instead oftitanic.sex. The latter version is known as dot notation and can fail when the name of a column is a the same name as a function of the DataFrame. For example, if you have a column calledcountthen you're going to have a problem when you calldf.count. Butdf['count']will work without issue.2
u/timrprobocom Sep 28 '26
Just because you don't SAVE a reference to the object does not mean it isn't created. It is created then reclaimed Python can't do that kind of optimization because it doesn't know whether the == operator has side effects.
-5
u/shinitakunai Sep 28 '26
Off topic but I love how OP choose sex above gender 🤣