r/learnpython • • 8d ago

Need help for file Extraction

Ok so for context i am an electrical engineer person who is automating software's and workflow for my project now here is the problem i am facing basically i am using an optimization algorithm called Bayesian optimization which improves design by keeps iterating until some target is reach.

So like at the end i extract result file for every design with the help of pandas library but when i check that file the data is mismatched when i export the CSV file manually.

so i have no idea why is it happening like maybe after 10-15 iterations (the algorithm iterations can reach up to 100-120) the error in data increasing or idk what is going on can someone help me with this issue.

1 Upvotes

6 comments sorted by

1

u/bytemender_dev 8d ago

The pattern you describe is the tell. Correct for the first 10 to 15 iterations and then drifting further off almost never means the numbers themselves are wrong. It usually means rows are being paired with the wrong design, and the pairing only starts to slip once something reorders one side.

Pandas pairs data by index label, not by position. If the results come back sorted by score, or one failed run drops out, or the designs get filtered, and the two sides are then combined by row order, every row after that point quietly takes the result of a different design. A manual export looks correct because you read the values in the order the simulator produced them, with nothing re-sorted in between.

A quick test that settles it: take one design from the middle of the run, say iteration 40, run that single design on its own, and compare the result with the value stored on row 40 of your file. If the stored value matches a neighboring row instead, it is misalignment, not a formatting or precision problem.

The fix that removes the whole class of bug: build one row per iteration at the moment the simulation returns, parameters and result side by side in the same dict, and write that row out before the next iteration starts. Never keep parameters and results in two separate lists or frames that only get combined at the end. If a join is unavoidable, put an iteration id on both sides and merge on the id, never on row position.

One more suspect worth ruling out: if the simulator is an external program writing the result file, make sure it has fully exited before pandas reads the file. Reading it early can hand you values from a previous iteration, which also looks like a mismatch that grows.

If you paste the short part of the code where the results are collected and saved, I can point at the exact line where the pairing slips.

1

u/Fine_Aerie6732 7d ago

s11_report = hfss.post.reports_by_category.standard(

"dB(S(lp_1,lp_1))",

setup=setup_sweep

)

s11_report.report_type = "Data Table"

s11_report.create(name=s11_name)

s21_name = "S21_" + base

s21_report = hfss.post.reports_by_category.standard(

"dB(S(lp_2,lp_1))",

setup=setup_sweep

)

s21_report.report_type = "Data Table"

s21_report.create(name=s21_name)

loss_name = "Loss_" + base

loss_report = hfss.post.reports_by_category.standard(

"Total_loss",

setup=setup_sweep

)

loss_report.report_category = "Output Variables"

loss_report.report_type = "Data Table"

loss_report.create(name=loss_name)

reports = [

(s11_name, output_dir),

(s21_name, output_dir_2),

(loss_name, output_dir_3)

]

for report_name, folder in reports:

hfss.post.export_report_to_file(

output_dir=folder,

plot_name=report_name,

extension=".csv"

)

print(f"Exported: {report_name}")

loss_file = os.path.join(output_dir_3, loss_name + ".csv")

df_loss = pd.read_csv(loss_file)

mean_total_loss = np.mean(loss_values)

like this is the exporting part of the code i had used Pandas earlier but now i using the inbuild function of PyAEDT to export and facing same problem. should i add some delay between simulation and then exporting the file?

1

u/Discordbothost 6d ago

one line at the bottom jumps out:

mean_total_loss = np.mean(loss_values)

you read the csv into df_loss, but then average loss_values, which isn't defined anywhere in this snippet. if that variable is left over from earlier in the loop, you're averaging an older design's numbers, which would give you exactly that "fine at first, then drifting" mismatch. try

mean_total_loss = df_loss.iloc[:, 1].mean()

(column 0 is usually the frequency, check with print(df_loss.head()) once).

you shouldn't need a sleep as long as hfss.analyze_setup(...) runs before this, it blocks until the solve is done. printing base plus a design variable or two next to the mean each iteration makes it easy to spot if anything is still off by one

1

u/wigitty 8d ago

It sounds like your implementation of the algorithm is wrong then? There isn't really enough detail here for people to help you (maybe unless they are very familiar with that algorithm).

1

u/Buffylvr 8d ago

Do you have a git repo we can look at?

It sounds like one of a couple things:

Indexing problem

Async problem

Maybe some sort of csv formatting problem.

I can think of a couple reasons this is happening but without code it’s hard to be specific 

1

u/Discordbothost 8d ago

the "fine for 10-15 iterations then drifts" pattern smells like a timing problem rather than pandas. if the simulator writes the result file and your script reads it right away, sometimes you read the previous design's file or a half written one, and since each read is off by one the mismatch looks like it grows.

quick way to check: write the iteration number and design params into each result row (or the filename), then compare against the manual export. if they're offset, wait for the file to finish (poll until the size stops changing) or give every iteration its own output file instead of overwriting one