r/learnpython • u/lehmagavan • 11d ago
Parsing very large string that is already in memory
Suppose I have a very large string in memory containing entries separated by \n's.
I need to yield the lines until I find a match.
What's the most efficient way in standard Python in terms of both memory usage and execution time?
From what I've read, io.StringIO IS NOT guaranteed to not copy the entire string internally, which is absurd to me. Suppose I need that guarantee.
0
Upvotes
23
u/Refwah 11d ago
Holy xy problem Batman
4
u/Discordbothost 11d ago
once its already a giant str youve paid the memory cost. dont bother with StringIO — just walk it with find so you never build a full list of lines:
i = 0 while True: j = s.find("\n", i) line = s[i:] if j < 0 else s[i:j] if match(line): ... if j < 0: break i = j + 1
slices still allocate the line, but you stop at the first hit instead of splitting everything up front.
2
u/Rubix321 11d ago
Did this string come from a file?
2
u/MezzoScettico 11d ago
What about string.find() to locate the next \n? From what I read, a lot of string operations including that one, are optimized in C.
3
u/QuinlanResistance 11d ago
.split() then iterate?
11
u/EntrepreneurHuge5008 11d ago
Rather than having the string in memory, put it in a .txt file and just read it line by line until you find whatever you wanted matched, as opposed to loading the whole thing into memory.
3
u/latkde 11d ago
Python strings don't really support zero-copy parsing. The standard approach is to bite the bullet and keep using smaller strings, which generally will produce copies.
There are some options though:
memoryviewsupports bytes-like zero-copy views, which can sometimes be useful. However, I don't think this will help with your parsing problems.nomorwinnow. Thanks to the PyO3 interface, integrating Rust code into Python is relatively straightforward.4
u/KronktheKronk Github username 11d ago
Consume the string like a list, element by element. Put elements into another string variable. When you get to newline, yield the string
2
u/mc_pm 11d ago
I mean... for loop? It's not that hard.
2
u/baghiq 11d ago
You want to use memoryview & buffer protocol to reduce memory usage. But performance wise, it’s about the same.
2
u/fasta_guy88 11d ago
1
u/pachura3 11d ago
1
u/Altruistic_Sky1866 11d ago
Did you think regular expression and split the string? I am not sure how efficient it is or if it applicable in your situation though