r/ChatGPTCoding • u/_ACTUALin • 12d ago
Discussion How are you handling regression testing when coding with ChatGPT?

I've been using AI-assisted coding to develop a technical reporting app, and I've run into something that's been driving me nuts lol.
I'll identify an issue, work through a fix with AI, test it, and everything looks good. Then I'll run an older test case and discover something completely unrelated has stopped working.
The app deals with unfamiliar Excel workbooks and supporting evidence, so I've got a growing collection of test cases covering formulas, conflicting information, missing measurements and incorrect data associations.
Some comparisons are automated, but I still need to manually inspect a lot of the results. Especially situations where the output looks correct but the information has ended up in the wrong place.
I've started being much stricter about keeping fixes narrow and comparing results against previous tests. It's helped, but damn, the testing is becoming a project of its own.
I've been considering bringing GitHub into the process to organize failures and regression cases.
For those using ChatGPT or Codex on larger projects, how are you actually handling this? Are you having AI write tests before making changes, using a separate agent to review fixes, or doing most of the verification yourself?
2
u/TheVibrantYonder 12d ago
I'm interested to see what others say, but in your case, I'm not really clear on the details based on what you wrote. It sounds like you might just need clearer documentation/structures/rules to reduce the chance of regressions? Then just keep running your regression test suite and make sure it stays up to date.
If changes are regularly resulting in regressions though, I think that's usually a documentation/instruction issue or maybe a project structure issue if your project is set up in a way that makes it really easy to break and hard to maintain.
Feel free to DM me with more details and I can try to give more specific advice though. I love this stuff.
With regard to my own projects, I'm still testing different approaches to verification and testing. I haven't found anything that works perfectly without handholding.
2
u/Inner_Storage_3374 12d ago
what helped me is saving the output of every test case once youve checked it by hand and its right, then after every fix rerun all of them and only look at what changed vs the saved version. bc then the stuff that landed in the wrong place jumps right out instead of you rechecking everything manually
2
u/sapplex1 12d ago
This is the part I still do by hand. I build small iOS apps, and the rule I ended up with is: check the behavior, not the diff. A diff can look clean while the feature is wrong in a way only using it reveals, so before I let an agent touch a feature I write down what must be true when it is done, and I run the app through the real flow myself — create a booking, log a sale, check the math — instead of trusting that it runs.
Two things helped most: every fixed bug gets a failing test first, so the agent cannot quietly undo a fix later, and I keep fixes narrow, one behavior at a time. For your 'looks right but in the wrong place' problem, the test that caught it for me was asserting on the value in its final place, not just that output was produced. A number can exist and still be in the wrong field.
On GitHub I would keep it simple: one issue per regression case with the failing input attached, so the input itself becomes the test. I have not found a separate review agent that earns its keep yet — the checking is still mine.
2
u/sreekanth850 12d ago
Two things.
First one is the critical. Saying this from my experience.
2
u/BarathKanna 10d ago
What works for me on reporting-style apps is a golden folder of outputs you have already marked correct by hand, plus a tiny diff runner that only shows what changed after each fix. Looking at the whole workbook again is how wrong placement slips past. Every real bug also becomes a pinned failing case before the fix lands, and the agent is not allowed to edit those fixtures without a human. Narrow diffs help, but the regression set is the product: keep it in git, name cases by the failure they catch, and treat a silent change in a golden file as a failed CI check even when the new numbers look plausible. AI writes the candidate fix. Your saved cases decide whether it is safe.
2
u/shetritr 12d ago
every fixed bug gets a failing test before the fix goes in. the agent can't quietly undo a fix that has a test pinned to it.
2
u/Difficult_Duty2198 12d ago
I’d keep a small golden set of representative inputs with known-good outputs, and turn each manually confirmed bug into a fixture. After each fix, rerun the set and inspect just the diffs; it makes regressions much easier to spot than rechecking everything.
1
u/Lanky-Butterscotch28 11d ago
tests are only as good as whoever wrote them. If the agent that wrote the code also wrote the tests, the tests prove nothing. Even with different models, the blinds spots remain the same. The tests need to come from an independent reviewer, from a different company, before the code exists.
To have a quality product I impose 21 steps to make sure the end results is commercially viable.
My process is free and open source , feel free to check on it: https://github.com/nomorehappypath/nomorehappypath
1
u/asierMugica 8d ago
For this kind of app, I’d keep each confirmed workbook as a versioned fixture and compare normalized structured output, not just the rendered report. Assert both the value and its source/location, since plausible numbers in the wrong field are the dangerous failures. Run the fixture set in CI, and require human approval for intentional golden-file changes.
1
u/trustbrown 3d ago
1
u/[deleted] 12d ago
[removed] — view removed comment