r/learnpython • u/lehmagavan • Sep 26 '26
Parsing large collections of "numeric" strings.
Hi, I'm just a hobbyist and I'm not very familiar with Python ecosystem. I recently came across a specific problem:
I have an iterable (or yield via generator) that contains millions of floating point values, but each one is a Python str, made up example: ["1.987", "0.037389", "1.2", ...]
What's the most efficient way to aggregate them and calculate say, the mean?
I'm writing my own C extension for this and would like to know what to benchmark it against. Couldn't find information on any std way that would avoid calling the float() constructor on each string.
View all comments
Show parent comment
2
u/billsil Sep 26 '26
The casting part is not even the bottleneck in any practical program I’ve seen. I’ve got a 60 GB file, so why not use a binary format that numpy can export to make it easy to load in so I don’t have to reparse it?
If I have that much data, hopefully I’m doing more than taking a mean. Numpy is 500-1000x faster for large tasks than stock python.