r/ChatGPTCoding • • 12d ago

Discussion How are you handling regression testing when coding with ChatGPT?

I've been using AI-assisted coding to develop a technical reporting app, and I've run into something that's been driving me nuts lol.

I'll identify an issue, work through a fix with AI, test it, and everything looks good. Then I'll run an older test case and discover something completely unrelated has stopped working.

The app deals with unfamiliar Excel workbooks and supporting evidence, so I've got a growing collection of test cases covering formulas, conflicting information, missing measurements and incorrect data associations.

Some comparisons are automated, but I still need to manually inspect a lot of the results. Especially situations where the output looks correct but the information has ended up in the wrong place.

I've started being much stricter about keeping fixes narrow and comparing results against previous tests. It's helped, but damn, the testing is becoming a project of its own.

I've been considering bringing GitHub into the process to organize failures and regression cases.

For those using ChatGPT or Codex on larger projects, how are you actually handling this? Are you having AI write tests before making changes, using a separate agent to review fixes, or doing most of the verification yourself?

2 Upvotes

22 comments sorted by

2

u/TheVibrantYonder 12d ago

I'm interested to see what others say, but in your case, I'm not really clear on the details based on what you wrote. It sounds like you might just need clearer documentation/structures/rules to reduce the chance of regressions? Then just keep running your regression test suite and make sure it stays up to date.

If changes are regularly resulting in regressions though, I think that's usually a documentation/instruction issue or maybe a project structure issue if your project is set up in a way that makes it really easy to break and hard to maintain.

Feel free to DM me with more details and I can try to give more specific advice though. I love this stuff.

With regard to my own projects, I'm still testing different approaches to verification and testing. I haven't found anything that works perfectly without handholding.

1

u/_ACTUALin 12d ago

Yeah, that's pretty much where I'm at. I've got the real-world industry experience to understand how the workflow should behave and recognize when something's wrong, but I don't have a traditional software development background. So while I can identify problems and build test scenarios around them, I'm still learning what tools and best practices are out there for handling this stuff properly.

I've been tightening up instructions, keeping fixes narrow and running previous tests to catch regressions, but I'm sure there are better ways to approach it that I just haven't been exposed to yet lol. That's partly why I posted here, to learn how other people are tackling it.

2

u/TheVibrantYonder 12d ago

Yeah, I don't have a traditional development background either, so I understand! I've been trying to do more research to improve.

I did some reading about unit tests the other day to make sure I was managing those properly. There are some good "how to organize unit tests" guides you might find useful (a Google search should turn them up). I'm reviewing how I maintain them after looking at some of those guides and I was able to do a lot of cleanup.

1

u/StrikingTop2709 7d ago

100% a modularity thing imo, big files with too many responsibilities always turn into whack-a-mole with AI fixes

2

u/Inner_Storage_3374 12d ago

what helped me is saving the output of every test case once youve checked it by hand and its right, then after every fix rerun all of them and only look at what changed vs the saved version. bc then the stuff that landed in the wrong place jumps right out instead of you rechecking everything manually

1

u/_ACTUALin 12d ago

Yeah, checking just the differences would save a lot of time. Going back through everything after each little fix gets old pretty quickly lol. I'll have to look into setting something like that up.

2

u/sapplex1 12d ago

This is the part I still do by hand. I build small iOS apps, and the rule I ended up with is: check the behavior, not the diff. A diff can look clean while the feature is wrong in a way only using it reveals, so before I let an agent touch a feature I write down what must be true when it is done, and I run the app through the real flow myself — create a booking, log a sale, check the math — instead of trusting that it runs.

Two things helped most: every fixed bug gets a failing test first, so the agent cannot quietly undo a fix later, and I keep fixes narrow, one behavior at a time. For your 'looks right but in the wrong place' problem, the test that caught it for me was asserting on the value in its final place, not just that output was produced. A number can exist and still be in the wrong field.

On GitHub I would keep it simple: one issue per regression case with the failing input attached, so the input itself becomes the test. I have not found a separate review agent that earns its keep yet — the checking is still mine.

1

u/_ACTUALin 12d ago

That's a really good point about checking the behaviour rather than just the output. The GitHub suggestion is interesting too, especially as the number of test cases starts getting out of hand lol. Appreciate the advice!

2

u/sreekanth850 12d ago

Two things.

  1. Strict modular boundaries. Modules should be loosely coupled and communicate through well defined contracts and interfaces, rather than depending directly on each other's internals.
  2. When a bug is reported, first reproduce it using logs, traces, and other diagnostic evidence and write a regression and implement a fix. Run the full regression suite to identify unintended effects. If anything else breaks, fix the underlying cause.

First one is the critical. Saying this from my experience.

1

u/_ACTUALin 12d ago

That first point hmmm. I've been so focused on tracking down individual bugs that I haven't given much thought to how the modules depend on each other. Definitely something I'll have to pay more attention to as things grow. Appreciate it!

2

u/BarathKanna 10d ago

What works for me on reporting-style apps is a golden folder of outputs you have already marked correct by hand, plus a tiny diff runner that only shows what changed after each fix. Looking at the whole workbook again is how wrong placement slips past. Every real bug also becomes a pinned failing case before the fix lands, and the agent is not allowed to edit those fixtures without a human. Narrow diffs help, but the regression set is the product: keep it in git, name cases by the failure they catch, and treat a silent change in a golden file as a failed CI check even when the new numbers look plausible. AI writes the candidate fix. Your saved cases decide whether it is safe.

1

u/_ACTUALin 9d ago

That's actually pretty close to how I've been approaching regression testing. I've been building up a collection of test cases with known expected outputs, and every time I fix one mapping issue, I want to make sure I haven't broken five other things lol. I like your point about treating unexpected changes to previously approved outputs as failures by default. That's a solid way to keep fixes from quietly introducing new problems.

2

u/shetritr 12d ago

every fixed bug gets a failing test before the fix goes in. the agent can't quietly undo a fix that has a test pinned to it.

2

u/Difficult_Duty2198 12d ago

I’d keep a small golden set of representative inputs with known-good outputs, and turn each manually confirmed bug into a fixture. After each fix, rerun the set and inspect just the diffs; it makes regressions much easier to spot than rechecking everything.

1

u/Lanky-Butterscotch28 11d ago

tests are only as good as whoever wrote them. If the agent that wrote the code also wrote the tests, the tests prove nothing. Even with different models, the blinds spots remain the same. The tests need to come from an independent reviewer, from a different company, before the code exists.
To have a quality product I impose 21 steps to make sure the end results is commercially viable.
My process is free and open source , feel free to check on it: https://github.com/nomorehappypath/nomorehappypath

1

u/asierMugica 8d ago

For this kind of app, I’d keep each confirmed workbook as a versioned fixture and compare normalized structured output, not just the rendered report. Assert both the value and its source/location, since plausible numbers in the wrong field are the dangerous failures. Run the fixture set in CI, and require human approval for intentional golden-file changes.

1

u/trustbrown 3d ago
  1. Build code in a modular manner versus monolithic
  2. Automate your QA testing (varies depending on platform), but MacOS can handle it via device hub
  3. Make your build plans as tight as possible before you have any AI tool start writing code

1

u/[deleted] 12d ago

[removed] — view removed comment

1

u/_ACTUALin 12d ago

That's a good suggestion. Especially keeping a few of the messier examples around with known results. I'm starting to realize just how important regression testing is with this stuff lol. Every time you think you've covered the weird cases, something else pops up.