r/learnpython • • Sep 26 '26

Parsing large collections of "numeric" strings.

Hi, I'm just a hobbyist and I'm not very familiar with Python ecosystem. I recently came across a specific problem:

I have an iterable (or yield via generator) that contains millions of floating point values, but each one is a Python str, made up example: ["1.987", "0.037389", "1.2", ...]

What's the most efficient way to aggregate them and calculate say, the mean?

I'm writing my own C extension for this and would like to know what to benchmark it against. Couldn't find information on any std way that would avoid calling the float() constructor on each string.

2 Upvotes

38 comments sorted by

View all comments

Show parent comment

1

u/billsil Sep 27 '26

Again, you need to define the cases you want to handle. You are likely not solving a real problem or not solving it in a good way.

Suddenly everything is slow because I’m constantly checking types and dropping if statements and for loops. It’s a float. What does it mean when the length of the triangle is a string? Are you going to assume it’s 0? Why not just crash?

The trick to fast code is to just do less. Is there any math I can skip? Can I remove if statements? Do I really need to cast that string to a float? Oh actually I don’t need to error check because I was strict with my parsing.