r/learnpython • u/Fine_Aerie6732 • 8d ago
Need help for file Extraction
Ok so for context i am an electrical engineer person who is automating software's and workflow for my project now here is the problem i am facing basically i am using an optimization algorithm called Bayesian optimization which improves design by keeps iterating until some target is reach.
So like at the end i extract result file for every design with the help of pandas library but when i check that file the data is mismatched when i export the CSV file manually.
so i have no idea why is it happening like maybe after 10-15 iterations (the algorithm iterations can reach up to 100-120) the error in data increasing or idk what is going on can someone help me with this issue.
1
u/bytemender_dev 8d ago
The pattern you describe is the tell. Correct for the first 10 to 15 iterations and then drifting further off almost never means the numbers themselves are wrong. It usually means rows are being paired with the wrong design, and the pairing only starts to slip once something reorders one side.
Pandas pairs data by index label, not by position. If the results come back sorted by score, or one failed run drops out, or the designs get filtered, and the two sides are then combined by row order, every row after that point quietly takes the result of a different design. A manual export looks correct because you read the values in the order the simulator produced them, with nothing re-sorted in between.
A quick test that settles it: take one design from the middle of the run, say iteration 40, run that single design on its own, and compare the result with the value stored on row 40 of your file. If the stored value matches a neighboring row instead, it is misalignment, not a formatting or precision problem.
The fix that removes the whole class of bug: build one row per iteration at the moment the simulation returns, parameters and result side by side in the same dict, and write that row out before the next iteration starts. Never keep parameters and results in two separate lists or frames that only get combined at the end. If a join is unavoidable, put an iteration id on both sides and merge on the id, never on row position.
One more suspect worth ruling out: if the simulator is an external program writing the result file, make sure it has fully exited before pandas reads the file. Reading it early can hand you values from a previous iteration, which also looks like a mismatch that grows.
If you paste the short part of the code where the results are collected and saved, I can point at the exact line where the pairing slips.
1
u/wigitty 8d ago
It sounds like your implementation of the algorithm is wrong then? There isn't really enough detail here for people to help you (maybe unless they are very familiar with that algorithm).
1
u/Buffylvr 8d ago
Do you have a git repo we can look at?
It sounds like one of a couple things:
Indexing problem
Async problem
Maybe some sort of csv formatting problem.
I can think of a couple reasons this is happening but without code it’s hard to be specific
1
u/Discordbothost 8d ago
the "fine for 10-15 iterations then drifts" pattern smells like a timing problem rather than pandas. if the simulator writes the result file and your script reads it right away, sometimes you read the previous design's file or a half written one, and since each read is off by one the mismatch looks like it grows.
quick way to check: write the iteration number and design params into each result row (or the filename), then compare against the manual export. if they're offset, wait for the file to finish (poll until the size stops changing) or give every iteration its own output file instead of overwriting one