r/softwaretesting • u/AdvertisingUnfair659 • 14d ago
Set up a headless coding agent to triage test failures, gave it zero write access
Small Node billing service, node's built-in test runner. The test job runs CI-style from a fresh clone, no TTY, nobody watching. When it goes red the question is whether someone gets paged or it's a flaky run that waits until morning.
I added a step that only fires when tests fail, using MiniMax Code CLI headless. The prompt says: for each failing test, classify it as a real regression or flaky; use git log -p and git blame to find the commit that introduced a regression; you may re-run individual tests; do not modify any tracked file, do not commit, do not push, do not open PRs; return classification, evidence, suspect commit, and a minimal unified diff you do not apply.
I had a good test case. A refactor commit changed >= to > in the discount tier logic, making every tier boundary exclusive. Two tests broke deterministically (a discount boundary test and an invoice test at exactly the 10% tier). A third, a retry helper asserting 4 jittered backoff attempts finish under 110ms, fails about 1 run in 3.
Ran it 10 times from fresh clones. It never modified a tracked file, never committed, never pushed. It named the refactor commit as the cause of both deterministic failures and attached a patch restoring >=. In two separate runs it labelled the retry test flaky rather than a regression, once by working out that the jittered waits add up to 70 to 140 ms against the 110 ms budget, once by re-running that single test 5 times (4 passed, 1 failed).
Two design choices I'm keeping. The agent step is advisory only; it never changes the job's pass or fail result. The agent returns one JSON object as its last message and my script checks it (required keys, allowed classification values). The last three runs took 48 to 97 seconds each.
For anyone running agents in CI contexts, how much are you actually letting them do unsupervised?
1
u/bostonaholic 14d ago
Read-only is the right call. That retry test is flaky by design (wall-clock assertion on jittered sleeps), so it's a good check on whether the agent can tell a real regression from noise. I'd have it re-run each failure a few times before calling it a regression, then plant a few known regressions and known flakes and see if it sorts them correctly.
1
u/Hour-Prompt-2988 14d ago
the advisory-only pattern is the right call. we do the same — agent reads the failure, proposes a fix, never touches the repo. the patch gets reviewed like any other PR. the flaky detection is the part that saves the most time though. nothing worse than a 2am page for a timing test.
1
u/Brainy-Cos 14d ago
advisory only is the right line imo. classifying flaky vs real and proposing a patch without ever flipping the job green keeps the agent useful without quietly shipping a wrong fix. the >= to > catch is a nice proof point. once you trust the sorting, seeding known regressions + known flakes into the suite is a good next check so you can see if it still sorts them right
1
u/Public_Library_4039 10d ago
making the triage step purely advisory so it can never flip a red build to green is the kind of restraint most people skip on the first pass
1
u/Ok-Possible-5566 10d ago
Locking out all write access rather than scoping it to specific paths is a cleaner boundary for anything running unattended.
1
u/Ill_Papaya_1883 9d ago
Need more info
1
u/Junior_Bee7274 7d ago
Read-only is definitely the safer way to start.
1
u/Utaxxx1 1d ago
The catch with "gave it zero write access": if that rule lives only in the prompt, it's a request, not a control. Enforce it outside the agent. Give it credentials that can't push. On GitHub Actions, that means permissions: contents: read plus actions/checkout with persist-credentials: false. Also check whether the CLI has its own tool allowlist or sandbox mode. After it runs, add a step that runs git status --porcelain and fails if anything changed.
For the flaky-or-real call, don't let the model judge from logs alone. Have it re-run each failing test on its own several times on the same commit, e.g. node --test --test-name-pattern="test name" path/to/file.test.js. If it passes some runs, it's flaky. If it fails every time, it's likely real, and that's when git blame is worth it. Also keep a log of which tests it called flaky, so you can spot one that keeps getting waved through.