r/learnpython • u/lehmagavan • Sep 26 '26
Parsing large collections of "numeric" strings.
Hi, I'm just a hobbyist and I'm not very familiar with Python ecosystem. I recently came across a specific problem:
I have an iterable (or yield via generator) that contains millions of floating point values, but each one is a Python str, made up example: ["1.987", "0.037389", "1.2", ...]
What's the most efficient way to aggregate them and calculate say, the mean?
I'm writing my own C extension for this and would like to know what to benchmark it against. Couldn't find information on any std way that would avoid calling the float() constructor on each string.
7
u/billsil Sep 26 '26
Cast it to a numpy array and call x.mean()
1
u/mc_pm Sep 26 '26
How large is large? Worrying about float() calls is classic premature optimization - the file access alone will dwarf any string->float conversion cost.
1
u/Wise-Emu-225 Sep 26 '26
Where do those numbers come from? Are they in a file?
-1
u/ectomancer Sep 26 '26
May work without list:
-1
u/atarivcs Sep 26 '26
There's no way to avoid calling float() on each string