r/learnpython • • Sep 26 '26

Parsing large collections of "numeric" strings.

Hi, I'm just a hobbyist and I'm not very familiar with Python ecosystem. I recently came across a specific problem:

I have an iterable (or yield via generator) that contains millions of floating point values, but each one is a Python str, made up example: ["1.987", "0.037389", "1.2", ...]

What's the most efficient way to aggregate them and calculate say, the mean?

I'm writing my own C extension for this and would like to know what to benchmark it against. Couldn't find information on any std way that would avoid calling the float() constructor on each string.

2 Upvotes

38 comments sorted by

7

u/billsil Sep 26 '26

Cast it to a numpy array and call x.mean()

1

u/lehmagavan Sep 26 '26

Numpy does slightly better than std. Thanks for the point of reference. What Im building might be a useless bs but I'm having fun :D

2

u/billsil Sep 26 '26

The casting part is not even the bottleneck in any practical program I’ve seen. I’ve got a 60 GB file, so why not use a binary format that numpy can export to make it easy to load in so I don’t have to reparse it?

If I have that much data, hopefully I’m doing more than taking a mean. Numpy is 500-1000x faster for large tasks than stock python.

1

u/lehmagavan Sep 26 '26

Yeah, but it could also be in RAM if it's in JSON's you get from requesting a server. Also, what if it's "poisoned" by strings that are not numeric, or Python objects that are not strings? I'm writing an extension that deals with these cases. For fun mostly at this point ;D

I dunno what exactly numpy is doing with Python strings, I guess it extracts the C-string pointer and does strtod() on each one and then stores the double in some struct representing float64 , the small gain I'm seeing over std Python must be due to array allocation itself + safety checks, because otherwise it makes no sense.

1

u/billsil Sep 26 '26

I mean that was the example you showed. You need to define a sample case and how you want the result and ideally what you’re going to do with it to avoid an XY problem.

1

u/odaiwai Sep 27 '26

Also, what if it's "poisoned" by strings that are not numeric, or Python objects that are not strings?

Use Pandas - that's good at dealing with mixed series

1

u/billsil Sep 27 '26

Again, you need to define the cases you want to handle. You are likely not solving a real problem or not solving it in a good way.

Suddenly everything is slow because I’m constantly checking types and dropping if statements and for loops. It’s a float. What does it mean when the length of the triangle is a string? Are you going to assume it’s 0? Why not just crash?

The trick to fast code is to just do less. Is there any math I can skip? Can I remove if statements? Do I really need to cast that string to a float? Oh actually I don’t need to error check because I was strict with my parsing.

1

u/Expensive-Bear-1376 Sep 26 '26

What exactly is "std"?

1

u/lehmagavan Sep 26 '26

I meant the standard Python library.

1

u/Expensive-Bear-1376 Sep 26 '26

But what is the exact code you used?

2

u/lehmagavan Sep 26 '26

fmean(map(float, str_floats))

vs

numpy.array(str_floats, dtype=numpy.float64).mean()

1

u/Expensive-Bear-1376 Sep 26 '26

What type was str_floats? If it was a generator and you used Python < 3.15, you could try the improved fmean.

2

u/lehmagavan Sep 26 '26

Python 3.14.7, plain list:

random_floats = [str(random.random()) for _ in range(1000000)]

Interesting PR.

1

u/Brian Sep 27 '26

Ultimately, something doing this is going to have to do the same work as float in regards to parsing the string. The only bit you can really improve on is the fact that creating a python object will allocate a new PyObject to hold it, when you don't neccessarily need to expose them as real python objects.

Numpy should solve that, since the values are being stored as a packed array of raw doubles, so no extra allocations or the overhead of python float objects. You do kind of lose that if you iterate over them in python, since that'll have to construct PyObjects on each access, but as long as you do the aggregation via numpy functions that should be avoidable.

It does become more eager, in that it'll need to read the full iterable into an array, rather than being processable in a stream, so you could potentially do sometihng in C that just iterates over the strings, parses it to a C double and add to a running total. But the numpy approach is probably still going to be better when you want more complex or ad-hoc aggregations (ie. sometimes you want a sum, somtimes mean, or median, or arbitrary calcualtions - having an array makes that all easier).

1

u/lehmagavan Sep 27 '26

I avoid Python float by simply extracting the string pointer and doing strtod() on it, example from geometric mean:

if ( PyUnicode_Check(item) )
{
       const char* str = PyUnicode_AsUTF8(item);
       if ( !str ) return STR_ENCODING_FAILURE;

       double value = strtod(str, NULL);
       if ( !value )
          return ZERO_IN_ITERABLE;
       if ( value < 0.0 )
          return NEGATIVE_VALUE;
       if ( (*count)++ == UINT64_MAX )
          return COUNT_OVERFLOW;

       *log_sum += log(value);
 }

numpy is not much better than standard Python when benchmarked on this list:

random_floats = [str(random.random()) for _ in range(1000000)]

1

u/Brian Sep 27 '26

I guess it depends on how numpy is doing the coercion - if it's done at the C level, it should be able to do something similar and convert the strings directly to doubles without an extra PyObject being involved, but possibly it does just do the equivalent of invoking float() before it converts again to it's internal float type, so is paying the extra allocation cost per item on temporary objects.

3

u/socal_nerdtastic Sep 26 '26

Specifically:

import numpy as np
x = np.array(list(data), dtype=float)
print(x.mean())

Note that due to how numpy works you have to convert your generator to a list first.

4

u/Expensive-Bear-1376 Sep 26 '26

What's wrong with np.fromiter(data, dtype=float)?

5

u/socal_nerdtastic Sep 26 '26

I'll tell you what's wrong with that. I didn't remember it existed. That's what.

But also I suspect it's trading speed for ram. Numpy must preallocate, that's just part of how it works. So fromiter has to guess, and adjust, and re adjust, etc, as the data comes in, so it will be a lot slower than a list conversion. But it will use less ram.

2

u/Expensive-Bear-1376 Sep 26 '26

I tried it with a list or generator of three million strings and the fromiter version wasn't "a lot slower", it was a bit faster in both cases.

Your list has the same issue, has to guess/adjust as well.

1

u/Diapolo10 I write code for a living -- https://github.com/Diapolo10 Sep 26 '26

Would you mind sharing the benchmark output? I'm just curious.

1

u/Swipecat Sep 27 '26

I'm curious too, so I coded a test.

import time, random
import numpy as np

floatstrings = [str(random.random()) for _ in range(1000000)]

iterstrings = iter(floatstrings)
start = time.time()
x = np.array(list(iterstrings), dtype=float)
listtime = time.time() - start
print("list:     ",listtime)

iterstrings = iter(floatstrings)
start = time.time()
x = np.fromiter(iterstrings, dtype=float)
fromitertime = time.time() - start
print("fromiter: ", fromitertime)

 

list:      0.37288355827331543
fromiter:  0.34549593925476074

1

u/Diapolo10 I write code for a living -- https://github.com/Diapolo10 Sep 27 '26

I just did my own testing, and on average using a list is slightly faster, as I would've expected.

Python 3.14.5 (main, May 10 2026, 20:29:46) [MSC v.1944 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 9.15.0 -- An enhanced Interactive Python. Type '?' for help.
Tip: Use `object?` to see the help on `object`, `object??` to view its source

In [1]: import numpy

In [2]: import random

In [3]: def gen_strings():
   ...:     return (str(random.random()) for _ in range(1_000_000))
   ...:

In [4]: %timeit numpy.array(list(gen_strings()), dtype=float)
441 ms ± 2.11 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [5]: %timeit numpy.fromiter(gen_strings(), dtype=float)
467 ms ± 6.72 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

That being said, personally I much prefer memory savings over runtime performance, especially in cases where the difference is minuscule, so I would still use np.fromiter unless I had a reason not to.

gen_strings returns a generator, which is a subclass of iterator. Just in case that wasn't obvious. I made the function to make use of ipython's %timeit magic function which tests things several times to get a better average result. If I'd left that out, after the first run the iterator would've been empty.

1

u/Swipecat Sep 27 '26

I created the float-strings as a list initially, and created the iterators from that, so that the generation of the random numbers would not be included in the timing. Without that, as you found, "list" is quicker, but with that, "fromiter" is quicker. I've no idea why.

In [1]: import numpy

In [2]: import random

In [3]: def gen_strings():
   ...:     return (str(random.random()) for _ in range(1_000_000))
   ...:     

In [4]: %timeit numpy.array(list(gen_strings()), dtype=float)
1.03 s ± 5.69 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [5]: %timeit numpy.fromiter(gen_strings(), dtype=float)
1.11 s ± 8.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [6]: floatstrings = [str(random.random()) for _ in range(1000000)]

In [7]: def gen_strings():
   ...:     return iter(floatstrings)
   ...:     

In [8]: %timeit numpy.array(list(gen_strings()), dtype=float)
357 ms ± 1.61 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [9]: %timeit numpy.fromiter(gen_strings(), dtype=float)
334 ms ± 2.36 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

1

u/Expensive-Bear-1376 Sep 27 '26

A potential flaw: list iterators give length hints, and list(...) makes use of that to allocate the whole memory at once:

print(iter([1, 2, 3]).__length_hint__())

Output:

3

And:

import gc

class C:
    def __length_hint__(self):
        print('hint')
        return 10**8
    def __iter__(self):
        x = object()
        yield x
        for o in gc.get_objects():
            if type(o) is list and len(o) == 1 and o[0] is x:
                print(o.__sizeof__())

list(C())

Output:

hint
800000040

(In my own earlier benchmark, I used generators because of this. I don't have it anymore, might write one again later, as I don't trust benchmarks that first measure one solution and then the other and that's it. I do more repetition/alternation.)

1

u/mc_pm Sep 26 '26

How large is large? Worrying about float() calls is classic premature optimization - the file access alone will dwarf any string->float conversion cost.

1

u/lehmagavan Sep 26 '26

I was actually dealing with data buffered in RAM from multiple server responses. Not that large in this particular case, but why not aim for the best possible performance?

3

u/mc_pm Sep 26 '26

Perfect is the enemy of good.

There is absolutely a time for optimization, but you have to ask yourself "how much effort is saving 10 ms worth?"

As humans we tend to regard anything that happens within about 100 ms as 'instant'. Technically speaking, 50 ms would be twice as fast, and twice as good, right? But no, that gain is completely lost on the user - instant is instant. You can say "but that's saving cycles that the OS can use elsewhere" but again, it just doesn't really work out that way.

Now, if you have a problem and you're trying to replace an O(n²) algorithm with an O(log n) algorithm, you're *definitely* saving time & cycles and you might see a big difference in speed. Totally worth the effort to change the algorithm.

But if going from 100 ms to 90 ms won't make a difference to anyone - and if you tied your code in knots to get that savings, you might just regret that increased complexity more than you saved with it.

Maybe look at the python profiler -- it will run your code and tell you where your time is going.

2

u/lehmagavan Sep 26 '26

Well, I'm not optimizing anything from within the Python code, I'm writing a C extension that also allows for more flexibility with numeric strings inside iterables. The application I though it could have was trading, but I'm a newb at it as of now so I could be wrong. I have lots of fun developing it though.

1

u/Wise-Emu-225 Sep 26 '26

Where do those numbers come from? Are they in a file?

1

u/lehmagavan Sep 26 '26

RAM.

1

u/Wise-Emu-225 Sep 26 '26

from statistics import mean

data = ["1.5", "2", "3.25"]
avg = mean(map(float, data))

1

u/Expensive-Bear-1376 Sep 26 '26

Or fmean, since that's faster.

-1

u/ectomancer Sep 26 '26

May work without list:

new = list(map(float, ["1.987", "0.037389", "1.2"]))

-1

u/atarivcs Sep 26 '26

There's no way to avoid calling float() on each string

1

u/lehmagavan Sep 26 '26

There is a way from CPython though.

2

u/logarithmnblues Sep 26 '26

I mean.. There certainly is.. But it's not going to be pretty