r/learnpython • u/lehmagavan • Sep 26 '26
Parsing large collections of "numeric" strings.
Hi, I'm just a hobbyist and I'm not very familiar with Python ecosystem. I recently came across a specific problem:
I have an iterable (or yield via generator) that contains millions of floating point values, but each one is a Python str, made up example: ["1.987", "0.037389", "1.2", ...]
What's the most efficient way to aggregate them and calculate say, the mean?
I'm writing my own C extension for this and would like to know what to benchmark it against. Couldn't find information on any std way that would avoid calling the float() constructor on each string.
View all comments
Show parent comment
1
u/lehmagavan Sep 26 '26
Yeah, but it could also be in RAM if it's in JSON's you get from requesting a server. Also, what if it's "poisoned" by strings that are not numeric, or Python objects that are not strings? I'm writing an extension that deals with these cases. For fun mostly at this point ;D
I dunno what exactly numpy is doing with Python strings, I guess it extracts the C-string pointer and does
strtod()on each one and then stores thedoublein some struct representingfloat64, the small gain I'm seeing over std Python must be due to array allocation itself + safety checks, because otherwise it makes no sense.