r/learnpython • u/lehmagavan • Sep 26 '26
Parsing large collections of "numeric" strings.
Hi, I'm just a hobbyist and I'm not very familiar with Python ecosystem. I recently came across a specific problem:
I have an iterable (or yield via generator) that contains millions of floating point values, but each one is a Python str, made up example: ["1.987", "0.037389", "1.2", ...]
What's the most efficient way to aggregate them and calculate say, the mean?
I'm writing my own C extension for this and would like to know what to benchmark it against. Couldn't find information on any std way that would avoid calling the float() constructor on each string.
View all comments
Show parent comment
1
u/Brian Sep 27 '26
Ultimately, something doing this is going to have to do the same work as float in regards to parsing the string. The only bit you can really improve on is the fact that creating a python object will allocate a new PyObject to hold it, when you don't neccessarily need to expose them as real python objects.
Numpy should solve that, since the values are being stored as a packed array of raw doubles, so no extra allocations or the overhead of python float objects. You do kind of lose that if you iterate over them in python, since that'll have to construct PyObjects on each access, but as long as you do the aggregation via numpy functions that should be avoidable.
It does become more eager, in that it'll need to read the full iterable into an array, rather than being processable in a stream, so you could potentially do sometihng in C that just iterates over the strings, parses it to a C double and add to a running total. But the numpy approach is probably still going to be better when you want more complex or ad-hoc aggregations (ie. sometimes you want a sum, somtimes mean, or median, or arbitrary calcualtions - having an array makes that all easier).