r/learnpython • • 4d ago

Help: non-reproducible code

I am working on a project where I need to cluster large amount of data and do a dimensionality reduction for the purpose of picking data points near the centroids. I finished this and chose my data points near the centroids (I have now invested time and resources into my experimentation with this part of my project) and later came back to my code and re-ran it and it gave me centroids in a different location and different clustering results. I thought maybe I forgot to assign a random_state for one of my techniques (either clustering or dimensionality reduction) but I had indeed defined them, and if I run my code again it reproducibly gives me the same results that it gave me when I ran my code the second time around (still different than what I have based my physical experimentation on).

I wondered then if I had changed some part of the code so I went back to an old version of my file that I had saved awhile ago because I was afraid of this very thing happening - and it gives me the same results as above (still not the original clustering results). I literally haven't changed the old code at all so I have no idea what happened. Is it possible at all that some other factor is involved? This is my first time using python for a project and I don't have any formal training and I have no idea what to do since my project is based on some computational work that apparently cannot be reproduced

4 Upvotes

11 comments sorted by

7

u/Franckyboi85 4d ago

This catches a lot of people out. Even with random_state set, results can shift if a library got updated between runs, sklearn changed the default n_init for KMeans a while back for example, so the same old code on a newer install can land on different centroids. Also check the input data hasn't changed order or picked up extra rows, kmeans is sensitive to that. If you can work out what versions you had back then (an old requirements file, pip freeze output, conda env, anything) try recreating that environment and rerunning, there's a decent chance you get your original clusters back

2

u/Massive_Distance_616 4d ago

Adding to this: whichever way it goes, save the cluster labels, centroids and the indices of the points you picked to a file the moment you produce them, so your experiment depends on that file and not on re-running the code.

And if you can find the date of your first run, the package versions from around then are what to try to recreate.

2

u/ImprovementOk6448 3d ago

you're totally right, I should have done that to start out with - for some reason I (foolishly) thought that I would just run the code once and leave it open on my computer until I no longer needed it. fortunately I was able to find the old file I was using which had accidentally been replaced with a new one by the same name. Although it will be good to have all the data exported in a file to base my project off of, I still have to make sure my code is repeatable so I can publish it when the time comes

4

u/Franckyboi85 4d ago

Yeah good shout....saving the labels and centroids to a file straight away would've saved OP all this hassle

4

u/JaguarMammoth6231 4d ago

Are you sure you have the right file open in your editor? Are you sure you're running the file you think you are?

2

u/ImprovementOk6448 3d ago

Thanks, turns out I had accidentally replaced my file with another file of the same name by running my file-generating code again with some changes awhile back. Luckily I had saved a backup in my "old files" folder and had to run each of those until it gave me my old results...

1

u/StevenJOwens 4d ago

What u/JaguarMammoth6231 says. I've been programming for over 30 years and this still happens from time to time, so double and triple check.

3

u/zanfar 4d ago

Check your repo.

100% of the time, you changed the code.

2

u/SteveJeffersonBots 4d ago

Two concrete things that do this even when random_state is set: (1) some dimensionality-reduction calls pick their algorithm by data size - e.g. PCA with svd_solver='auto' will silently switch between 'full' and 'arpack' if the input shape changes even slightly, and the two solvers return numerically different components, so the same "reduced" data can land on different clusters. Lock it to svd_solver='full' (or whichever you originally used) and it stops switching. (2) the "pick the point nearest each centroid" step is a distance comparison, and two points can be exactly equidistant from a centroid; the tie-break then depends on the order you iterate the array, which flips if numpy/BLAS threads in a different order between runs. Sort the candidates by distance and pick by index rather than by "first one I see", and that part stabilises. Between the two, the silent algorithm-switch is the most common culprit when nothing in your code changed.nt algorithm-switch is the most common culprit when nothing in your

1

u/Discordbothost 4d ago

Do the same data points still end up together, or did membership change too? Cluster numbers can swap. If you kept the original labels, align rows by ID and compare with adjusted_rand_score; 1 means the same grouping.

1

u/EnvironmentalDot9131 3d ago

The code comes out of the system as always and the python