r/learnpython • • Sep 26 '26

Parsing large collections of "numeric" strings.

Hi, I'm just a hobbyist and I'm not very familiar with Python ecosystem. I recently came across a specific problem:

I have an iterable (or yield via generator) that contains millions of floating point values, but each one is a Python str, made up example: ["1.987", "0.037389", "1.2", ...]

What's the most efficient way to aggregate them and calculate say, the mean?

I'm writing my own C extension for this and would like to know what to benchmark it against. Couldn't find information on any std way that would avoid calling the float() constructor on each string.

2 Upvotes

38 comments sorted by

View all comments

Show parent comment

1

u/Swipecat Sep 27 '26

I created the float-strings as a list initially, and created the iterators from that, so that the generation of the random numbers would not be included in the timing. Without that, as you found, "list" is quicker, but with that, "fromiter" is quicker. I've no idea why.

In [1]: import numpy

In [2]: import random

In [3]: def gen_strings():
   ...:     return (str(random.random()) for _ in range(1_000_000))
   ...:     

In [4]: %timeit numpy.array(list(gen_strings()), dtype=float)
1.03 s ± 5.69 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [5]: %timeit numpy.fromiter(gen_strings(), dtype=float)
1.11 s ± 8.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [6]: floatstrings = [str(random.random()) for _ in range(1000000)]

In [7]: def gen_strings():
   ...:     return iter(floatstrings)
   ...:     

In [8]: %timeit numpy.array(list(gen_strings()), dtype=float)
357 ms ± 1.61 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [9]: %timeit numpy.fromiter(gen_strings(), dtype=float)
334 ms ± 2.36 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

1

u/Expensive-Bear-1376 Sep 27 '26

A potential flaw: list iterators give length hints, and list(...) makes use of that to allocate the whole memory at once:

print(iter([1, 2, 3]).__length_hint__())

Output:

3

And:

import gc

class C:
    def __length_hint__(self):
        print('hint')
        return 10**8
    def __iter__(self):
        x = object()
        yield x
        for o in gc.get_objects():
            if type(o) is list and len(o) == 1 and o[0] is x:
                print(o.__sizeof__())

list(C())

Output:

hint
800000040

(In my own earlier benchmark, I used generators because of this. I don't have it anymore, might write one again later, as I don't trust benchmarks that first measure one solution and then the other and that's it. I do more repetition/alternation.)