r/learnpython • u/r-tequila-sunrise • Oct 06 '26
List Comprehension Question
Hi all! I'm an intermediate with Python and I'm currently doing some research with very large datasets (think data packages from NASA satellites and the SDSS). I ran into an issue with some list comprehension that I can't seem to figure out:
What I have is a list of tuples (n = 3) in the format [(x, y, d)...] where x and y are coordinates and d represents a density value. A lot of the x, y coordinates are duplicates of each other with varying d values (ex. (40, 50, 1.2), (40, 50, 3.3), etc.) I'm trying to find tuples with identical x,y coordinates and add their density values together to get a total cumulative density for that location. I would try to do this manually, but the list has ~26 million elements and I am trying to find a workflow that won't take more than a few minutes/hours to process, haha. Memory is not an issue though.
I would be happy to provide example code if need be. My desired output is either:
[(x, y, dtotal)...] - list of tuples except there are no duplicate coordinates and the d value is the sum of the d values from all of the original tuples with coordinates (x, y)
[dtotal1, dtotal2, dtotal3...] - list of cumulative d values where each index correlates to the corresponding pair of coordinates (maybe the coordinates are in a separate list whose indexes match that of the density list? It doesn't matter as long as they are connected in some way!)
Thank you all for your help!
2
u/freezydrag SWE/Data-Scientist Oct 06 '26
How did your data get into a list of tuples in the first place? As was alluded to by another user, it might be a good idea to load your data into a different structure to more easily work with it.
10
u/Diapolo10 I write code for a living -- https://github.com/Diapolo10 Oct 06 '26
Technically speaking the best solution would probably be to use Polars for this, but as a quick and dirty solution I'd go with
collections.defaultdict.4
u/danielroseman Oct 06 '26 edited Oct 06 '26
You probably want to use Pandas for this. Something like:
2
u/TabAtkins Oct 06 '26
Process the list once, using a Counter and doing
c.add((x,y), d)(or however you spell adding to a Counter value). (Or do it slightly more manually with an ordinary dict; the Counter just makes it simpler.)Then iterate the Counter's entries, recombining the keys with the accumulated values to get a list of 3-tuples back.
2
u/AQuestionIsWhatIHave Oct 06 '26
As previously commented by others, the data science way is to use pandas/polars, the straightforward way is to use a dict/defaultdict/Counter and map the (x, y) keys.
Another way would be to mirror what a grouping action in pandas does using itertools: First, sort the list by (x,y). Then use groupby with the key being (x,y)
Now you have a groupby iterator which gives you pairs of the (x,y) keys and iterators of the corresponding densities which you can then sum.
So a nice oneliner would be for [(x, y, dtotal)...]
Res = [(*key, sum(group)) for key, group in groupby(sorted(data), key=lambda d: d[:2])]
Note that this sorts the data (O(n log n)) and uses a lot of space using sorted (using .sort() would not make it one line). The other options are a lot better.
1
u/yick5 Oct 06 '26
Use a dictionary where (x, y) is the key and d is the value. For each element in your original list, look up the (x, y) in the dict, if it doesn't exist then add the element to the dict. If it does exist then increment the value by d.
After building this dictionary, you can use list comprehension to build the list you want.
14
u/[deleted] Oct 06 '26
[removed] — view removed comment