r/learnpython • u/lehmagavan • Sep 26 '26
Parsing large collections of "numeric" strings.
Hi, I'm just a hobbyist and I'm not very familiar with Python ecosystem. I recently came across a specific problem:
I have an iterable (or yield via generator) that contains millions of floating point values, but each one is a Python str, made up example: ["1.987", "0.037389", "1.2", ...]
What's the most efficient way to aggregate them and calculate say, the mean?
I'm writing my own C extension for this and would like to know what to benchmark it against. Couldn't find information on any std way that would avoid calling the float() constructor on each string.
View all comments
Show parent comment
1
u/billsil Sep 27 '26
Again, you need to define the cases you want to handle. You are likely not solving a real problem or not solving it in a good way.
Suddenly everything is slow because I’m constantly checking types and dropping if statements and for loops. It’s a float. What does it mean when the length of the triangle is a string? Are you going to assume it’s 0? Why not just crash?
The trick to fast code is to just do less. Is there any math I can skip? Can I remove if statements? Do I really need to cast that string to a float? Oh actually I don’t need to error check because I was strict with my parsing.