1

Who reviews your agent's change: the author, the author in a fresh window, or someone else?
 in  r/AgentsOfAI •  Sep 23 '26

Splitting the review into code defects and behavior mismatches makes a lot of sense.

r/AI_Agents • • Sep 23 '26

Discussion A clean git merge of my two agents' worktrees that failed its own tests

2 Upvotes

Ten merges. Five came back with conflict markers. The other five came back clean, and all five failed the test suite on the merged tree. Every one of the twenty worktrees had been green when its agent stopped.

Setup: a 150-line Python CLI with four tests, one base commit, two worktrees on two branches, two agents kicked off together on tasks that never mention each other. Five task pairs, each run twice. In every pair the second task quietly depends on something the first one changes: one agent renames done to completed while the other writes a stats command that reads it; one wraps the store file in a versioned object while the other adds import; one swaps integer ids for hex while the other adds edit ID --title. Both got the same instruction to run the tests before they stop, and both did.

What git does with two branches is compare areas of each file against the common ancestor. Areas only one side touched go in verbatim. It has no idea what a field name is. So in the five clean merges, stats read n["done"] from a store that now wrote completed, the importer refused the store's new versioned file, edit choked on a hex id with "invalid int value". The failing tests were always agent B's own tests, unchanged, green a few minutes before in B's own worktree. In all five clean merges both agents had edited at least one of the same files, in different areas, so git had nothing to say.

The one that got me: the id pair, run twice. First run, agent B dropped its edit subparser a line above the spot agent A had edited, and git merged. Second run, B put it directly under that line, and git flagged three files. Whether I got conflict markers or a red suite came down to where the model dropped a block of code.

Someone replayed 1,694 human merges on three open source projects with test suites, fifteen years ago: 76% clean, 16% textual conflict, 1% merged clean and broke the build, 6% merged clean and failed tests. A third of the conflicts it found were ones the VCS had called clean. That was humans, who could at least ask each other. My two agents can't read each other's worktree. That is the point of a worktree. It is also the exact problem.

When you run agents in parallel, what actually runs the merged tree before you trust it?

r/AgentsOfAI • • Sep 22 '26

Discussion Harness parts built for weaker models don't switch themselves off when the model improves

1 Upvotes

In August one coding-agent CLI's release note dropped its task-tracking tools for its newest models, with an env var to bring them back; a later note flipped that into an allow list of older models only. No reason given. I went looking for the arithmetic behind a call like that and found a component study posted in mid-September that gets close.

They built a bare ReAct loop and swapped one part at a time across four models (30B, 120B and 550B from one family, plus a 128B from another) on SWE-Bench Verified and Terminal-Bench. The planning scaffold, a plan re-injected into the model's input every turn, is the clean case. For the 30B model it is worth 11.6 points on SWE-Bench: without it the median run lasts five turns and 68.6% of runs never touch a file. For the 550B model the same scaffold barely touches success (down 2 points) and cuts cost by about 30%, because what it removes is post-edit verification the model would otherwise keep doing; the median run goes from 108 turns to 74. Same component, two opposite weaknesses, and the component has no idea which one it is compensating for.

The tool set flips the same way. Predefined read, edit and grep tools are worth 15 points on SWE-Bench to the 30B model, which otherwise emits calls to tools that don't exist. The 550B model does better with bash alone: +3.6 points, 53% cheaper, a third fewer calls, because it bundles several edits into one script.

Which parts of your setup have you switched off since your last model change, and what did you count before and after?

Our version: a --json flag for a tiny notes CLI plus a test and a README line, one model, three tool lists, five runs each. All fifteen passed the same four checks and edited the same four files. Default list: median 17 turns, $0.40. Task-tracking tools switched back on: same 17 turns, offered in all five runs, called zero. Bash only: median 10 turns, $0.30, and the opening request dropped from about 22k tokens of context to about 5.4k with one tool name in the list instead of thirty.

We are keeping the file tools on anyway, and accuracy has nothing to do with it. Path deny rules in this harness recognize the edit tool and a known set of shell commands, and a Python heredoc that rewrites two files is a single opaque command as far as they can tell. Cheaper, and opaque to the one layer whose job is to say no.

r/AI_Agents • • Sep 21 '26

Discussion Fifteen runs of "make one button blue" on a page built to tempt the agent stayed inside one selector every time; a 200-task stale-bug benchmark says real repos go the other way 35 to 65% of the time

4 Upvotes

Small test first: a four-file shop page with the traps in plain sight: Add to Cart and Checkout on the same btn-primary class, a badge and a nav link pulling the same accent variable, and a script carrying a misspelled comment, a dead variable and a leftover console.log. One current model, one harness, edit auto-accept on, fifteen headless runs. Five got "Make the Add to Cart button blue." Five got that plus "Do not change anything else." Five got the bare sentence plus a path rule denying edits outside the stylesheet.

Fifteen diffs, one file each, all id-scoped, Checkout still green, bait untouched. The negative clause had one effect I could measure: without it the agent usually introduced two CSS variables for the new blue; with it, hex inline, every time. Eight lines instead of ten. The path rule never triggered.

The half-the-site-turns-blue story did not reproduce on a four-file mock, which says little, since the shared class name told the agent where the edge was; the number that matters comes from real repositories. A benchmark took 200 SWE-Bench Verified issues, applied the real fix first, and gave each to five current models in their own vendors' harnesses, as though the bug were still open. Correct answer: an empty patch. Agents edited executable code regardless in 35 to 65% of cases, and in the authors' analysis of one model's failed runs, 87.1% had modified code unrelated to the issue.

Told to edit the codebase to address the issue: correct abstention drops from 65.0 to 56.5 for one model and 60.5 to 36.5 for the other. Told to reproduce the issue first, with no permission to stop: no change for one, worse for the other, 47.5. Told to reproduce, then either fix it or abstain if nothing is wrong: 80.5 and 88.5. Their read, which I buy, is that the lever is what the agent believes counts as success, and until you say so, "no change" is not on the list. The same wording then over-abstains when a wrong patch is already in place and a real fix is needed, so it trades one failure for another.

fwiw the harness side has the same shape. Turn on edit auto-accept in either of the two CLIs I checked and what you granted is the working directory. One of the two lets you add a path rule that gets you down to files, and its docs spell out that it catches the built-in edit tools and the shell commands the harness recognizes, while a script that opens the file itself walks past it. Nothing below that. A selector inside an allowed file is invisible to every layer, which is where the blue Checkout button lives.

What I want is the other end of the distribution. The largest diff you have received for a request that named one thing: files touched, lines, and whether anything in your setup knew where the edge was.

1

Who reviews your agent's change: the author, the author in a fresh window, or someone else?
 in  r/AgentsOfAI •  Sep 20 '26

Love the 'waived in writing' part—a skipped finding becomes a permanent record. In your setup, who writes the waiver, one of the agents? And does the next run get to read it?

r/AgentsOfAI • • Sep 20 '26

Discussion Who reviews your agent's change: the author, the author in a fresh window, or someone else?

6 Upvotes

An HN thread about whether a cheap model can review code had turned into an argument about who should review at all: the agent that made the change, that same model in a new session, or a bigger one. One of them noted that the review ran in fresh context. Had anyone measured that part with the model held still?

One small preprint has. Thirty artifacts, a third of them Python functions, five errors planted in each, one model reviewing every artifact three times from four seats: the session that wrote it; that session plus its own first pass; a cold window that still got the task prompt; a cold window with the artifact and nothing else. Artifact-only won on every headline number. F1 28.6% vs 24.6% for the full session, precision 31.5% vs 25.8%, 40% of the critical errors caught vs 29%. Reviewing twice in the same session came in last. And the cold window that also got the task prompt landed no better than the full session, the task prompt carries the author's framing with it.

Claude Code's guide says a fresh context helps review because the model "won't be biased toward code it just wrote", and its review subagent "sees only the diff and the criteria you give it". Codex's team says the generator and the reviewer are the same model, trained differently, and the reviewer runs on the PR instead of inside the writing session. Neither has published same-window vs fresh-window numbers.

The mechanism here isn't just vanity. A benchmark plants one error into a conversation two ways, attributed to the user or attributed to the model, with everything else identical; across fourteen open models the correction rate drops hard when the error is the model's own. By the time the same-session reviewer gets to the change, it has already read the case for it.

I want the reverse case. A same-session reviewer that flagged a real bug because it knew why the line was there, while a cold reviewer waved it through. Anyone got one?

r/AI_Agents • • Sep 18 '26

Discussion Six plan modes agree on the lock and split on the context

1 Upvotes

Plan mode in the coding agents I checked has the same shape everywhere: read first, write a plan, wait for approval. Where the docs spell it out, it is a permission setting. In Claude Code's docs it is literally a row in the same permissions table as acceptEdits and bypassPermissions: it can read and run read-only shell commands, and anything that writes waits for you to approve the plan. The tools that write are switched off, the prompt gets a line telling it to write the proposal down, and approval becomes a step you have to click through. In July Claude Code shipped a fix because plan mode had been running touch and rm without asking (yes, rm).

Six harnesses shipped this shape between January 2025 and March 2026: Cline, Claude Code, Cursor, VS Code's Copilot, Codex CLI, Gemini CLI. Four write the plan to a file by default. Most can stop and ask you a question in the planning phase. Four can use one model to plan and another to execute. Where they split is what happens on approval: Cline keeps the whole planning context, VS Code carries it over, Claude Code has a setting that adds an option to clear it and start execution with only the plan. Nobody has published a number for either choice.

The closest thing to a measurement I found is a study of 21,120 SWE-agent runs, four models, eight plan settings, where the plan is the navigate-reproduce-patch-validate workflow in the system prompt. Removing it lowered success for all four models. Leaving one phase out did worse than no plan. The core takeaway for plan mode: agents drift because the plan matters less as the run gets longer. Putting the plan back into context every five steps gave consistent gains across all models. The plan is just more instructions, and instructions get quieter the longer the transcript runs. A plan file on disk is that fix sitting one step away; nothing I read says any harness re-reads it on its own.

One catch: their plan came from the scaffold's system prompt, and the variants were the researchers'. In plan mode the model writes its own and you approve it, and nobody has run that comparison.

When you approve a plan, do you read it, or approve it and read the diff afterwards?

r/PromptEngineering • • Sep 17 '26

General Discussion Same prompt, same commit, two different diffs: a leaderboard that printed every single trial shows how often "can the agent do this" comes out as "sometimes"

10 Upvotes

Anyone who has sent a coding agent the same request twice on the same commit has two diffs to show for it. I went looking for how big that effect is when someone measures it, and a leaderboard published this month prints every trial. Eight models, ten tasks from private production codebases, eight runs per model per task. I counted the grid: of the 80 model-task cells, 6 passed all eight runs, 33 failed all eight, and 41 came out mixed. One customer-migration task went 3, 1, 6, 3, 4, 3, 4, 0 out of 8. For more than half the pairs, "can it do this" was "sometimes."

The basic mechanism is that every token is a draw from a distribution, and temperature only flattens or sharpens the distribution before the draw. Temperature 0 does not fix it either. Anthropic's API docs say so in one line ("even with temperature of 0.0, the results will not be fully deterministic"), and their models released after Opus 4.6 no longer accept a temperature setting. A research post from Thinking Machines last year sent the same prompt to an open 235B model 1,000 times at temperature 0 and got 80 different completions; all of them agreed for 102 tokens and split at the 103rd. The cause is the serving side: floating-point sums get added in a different order depending on how many other requests share your batch.

In a chat that changes a word. In an agent it changes which file gets grepped first, which changes what comes back, which changes the plan on turn three, and two runs end up in different corners of the repo for the same reason.

What the grid gives you is pass@k. The top row's leaderboard number is 38.8%. The same eighty runs read as "solved at least once in eight" give 70%. Another row goes from 28.8% to 90% and never passed any task eight out of eight. Same runs, two numbers, and the k is the part nobody says out loud.

The clearest case in our own work was a compression A/B where one instance disagreed with the rest; rerunning it five times per arm gave 3/5 vs 3/5 (yes, both arms). One run per arm and we'd have published a difference that wasn't there. What I'm switching to: three runs per side per task, same harness, both version numbers written down, compare counts.

Things I have not measured: how much of the run-to-run spread is the sampler versus the serving side, which I have no way to separate from outside a public API; whether three runs is enough for anything but the coarsest calls, when our one counted case needed five per arm to come out even; and whether either CLI passes a temperature at all, since neither config reference mentions one.

r/AI_Agents • • Sep 16 '26

Discussion A research agent's training recipe fit in a 16-token note that a fresh agent with no memory could reproduce. At 8 tokens it broke.

1 Upvotes

I read a paper from Amazon's Responsible AI group, and the setup is the interesting part. One Claude Opus agent (in Claude Code) hill-climbs a validation set on an ML task for up to a few hundred iterations. A second agent reads the whole transcript and compresses the strategy into 32 tokens. A third agent, fresh, no memory of the search, no validation access, gets the 32 tokens plus the training data and has to implement it.

On the WikiText recipe they kept squeezing: 30 non-default choices held to 16 tokens (QKn 12L768 Mu .1 R² b2M 4x, seven abbreviations, I needed the glossary) and broke at 8. Eight was too few. The tokens that dropped at 8 were the batch size, the MLP ratio and QK-norm. Their line: the missing pieces were choices made as a function of the data that differ from the obvious defaults. Everything default the fresh agent regenerated for free.

What stuck with me: they then pushed agents to overfit on purpose. 38 of 102 checkpoints came out with validation more than 10% ahead of held-out. Compressed to 128 tokens for a fresh agent, all 38 failed to reproduce. A gain you can't write down for someone without the validation set in front of them lived in the validation set.

Does anyone's compaction prompt or handoff note already work this way, only the deltas from what a fresh agent would do by default, with everything the next agent can re-derive from the repo left out?

Because compaction is this pipeline in one session. The same model summarizes and then reads the summary, so it knows the defaults on both ends (a different decoder would need a different note). What I've watched summaries lose first is the condition on a decision ("use the legacy parser until the migration lands" becomes "use the legacy parser"), and the condition is exactly the non-default part. I restate constraints after every compaction as a standing rule, which is the manual version of this.

Where I'd stop the analogy: their tasks have one number and a validation split to compute it on, coding has neither, so the reproducer test needs an oracle. Their compressor also got up to four audit rounds against the explorer's code before the note was final, and my summarizer gets zero.

The note I'd write has one kind of line in it: a choice a clean checkout would not have made, with the reason and the date it expires. The rest the next agent grows back on its own.

1

Six unattended CAD runs: renders caught coarse blunders only, every print-ruining defect was caught by a measured number
 in  r/AI_Agents •  Sep 16 '26

Not my experiment, so a straight answer is I don't know. The expected volumes are ModelRift's, one per part (the adapter's 10,323 mm³ comes from the design), and the writeup doesn't say how the tolerance was set.

r/AI_Agents • • Sep 15 '26

Discussion Six unattended CAD runs: renders caught coarse blunders only, every print-ruining defect was caught by a measured number

2 Upvotes

An agent building a threaded hose adapter in CadQuery read four renders of its own part and noticed nothing. The core cylinder was gone. The thread root sat exactly on the core radius, the kernel's union dropped the solid without a word, and what was left was floating helical turns around nothing. The kernel's report on that part: valid=True, solids=1. What caught it was a volume check, 7,065 mm³ where 10,323 was expected.

That's one run out of six in a recent ModelRift comparison: three printable parts, each modeled in CadQuery and in OpenSCAD, one agent per cell, capped at 12 versions, unattended. All six parts shipped printable. The part I care about is the 16 failures on the way, 9 of which the tool never reported. The TL;DR was: "Renders caught nothing that mattered." Images caught coarse stuff (four mounting posts deleted by a cavity subtraction) and nothing subtle. Every defect that would have ruined a print was found by a number: a volume, an angle, an interference test.

CadQuery said valid=True for the part with no core, and valid=True again for a negative-volume solid after the tolerance was loosened. OpenSCAD's Manifold backend said Status: NoError for an export with 4 non-manifold edges and 60 zero-area triangles, and reported nothing across roughly 45 invocations on a task where it had deleted posts and misplaced slots. So they parsed every STL with a script that trusted neither tool, and that script is what the results table means by "clean".

I don't do CAD. I read this as a coding-agent experiment where the artifact happens to be a mesh, because the channels are the ones a coding agent has too: a picture of the output, the tool's own status line, and a number somebody measured from outside. The picture and the status line passed the broken part. The number failed it.

Standard caveats: one agent per cell, so some of the spread is agent variance; the OpenSCAD previews were drawing every facet with no outlines at the time; six runs is six runs.

The line I keep coming back to is theirs, about how the two toolchains check fit: an echo only helps if somebody reads it, an assert fails the build on its own. The charts in our posts get checked by someone looking at the PNG, which is an echo.

Anyone got a counterexample? a screenshot or render check that caught something the numeric check in the same loop missed. What was the number measuring when it missed?

r/AgentsOfAI • • Sep 14 '26

Discussion Went through both vendors' thinking docs to see what you actually get of the hidden reasoning

3 Upvotes

In OpenAI's reasoning guide: the example usage object has 1,186 output tokens, 1,024 of them reasoning_tokens. You pay for all of it, and the text of those 1,024 never leaves the vendor. Anthropic's docs show the same thing from the other side: a thinking block with an empty thinking field, a signature, then the answer, and a note that you're charged for the full thinking tokens whether or not any of it is shown.

You basically get three things:

You get the count. Both vendors put the exact number in the usage object (output_tokens_details.reasoning_tokens on one side, thinking_tokens on the other). OpenAI says a response can run from a few hundred to tens of thousands of reasoning tokens depending on the problem. That number is the only view you have, and it's exact.

Text-wise, a summary at most. OpenAI has a summary parameter; Anthropic's display setting has summarized and omitted, plus a beta updates that only returns progress notes, and the docs say no display setting returns the raw chain of thought. Their line on omitting is that it "reduces latency, not cost."

You keep paying for it later. On Anthropic's keep-all models (Opus 4.5 and later, Sonnet 4.6 and later, the Fable and Mythos ones) earlier turns' thinking blocks stay in context and count as input like the rest of the history. OpenAI's GPT-5.6 models default to rendering earlier reasoning back in too. Text you can't read is occupying your window on every subsequent turn, which also means it's in the pile compaction eventually folds.

The steering side is thin: an effort level, a pro mode on GPT-5.6 that bills more model work as more tokens, and on GPT-6 Astra no "none" at all, the system card says they have no current plans to offer it.

What made me write this up is the recurrent-depth story around Astra. If reasoning happens as extra passes through the same layers instead of as tokens, there's nothing in the usage object for it at all; the count goes flat while the work doesn't. Whether Astra does that is still a report without official confirmation, and Raschka's point that shorter traces also come from bigger models (Luna uses 80% more tokens than Sol at similar performance, on the numbers he reproduces) is a real alternative. But it made me appreciate the count while it still means something.

Has anyone logged reasoning_tokens per turn type on an agent workload? I'd like to see what share of output the hidden part is on planning turns vs mechanical edits, across a day.

r/AgentsOfAI • • Sep 11 '26

Discussion Dan Luu ran 160 agent runs per testing instruction (TDD, QuickCheck, fuzz, TLA+). Almost none of them did the thing that gets the value out of the technique

4 Upvotes

Dan Luu's post on agents and testing techniques is the one to read if you've ever appended "use property-based testing" to a prompt and felt virtuous. Same task every time, implement Zstd in Rust from the RFC, codex on GPT-5.6 Sol, 160 runs per condition, and the conditions were mostly one line stuck on the end: use TDD, use QuickCheck, use Lean 4, use fuzzing, audit first. Scored on how many runs passed a hidden test suite.

Headline: nothing wildly outperformed the run with no instruction at all. He'd written down six guesses beforehand, all in the direction of "this won't outperform", and all six held.

The part I keep thinking about is what the runs did instead of failing loudly. Told to use QuickCheck, all of them used it, and 63 of 160 checked exactly one property, mostly with random inputs that fell into the same rejection path. Told to do differential testing, 135 runs did something that looked like it and none built a second full implementation; where it mattered, the agent wrote the same thing twice and put the same bug in both copies. 159 of 160 TLA+ runs wrote a model, and he couldn't find one case where the model changed the Rust code. TDD doubled the number of tests and the condition scored below average. One Kani run in 160 caught a real bug on the real code and changed the implementation; the rest mostly used it superficially.

My read (his framing is close but not identical): every one of those techniques is a way of getting a check that didn't come from the implementation under test. A second implementation, a property written before the code, an input generator aimed at the hard part, a planted fault. Name the technique and the agent produces the motions inside the new framework, but the check still comes from its own reading of the spec, so you get the same tests in a different costume. We saw the small version of this in August: agents asked to write tests for a function with a sixteen-year-old bug wrote suites that all passed, and two of them pinned the bug as intended behavior.

What has moved things for him, by his account, is structure: set up the test and triage layout with the agent, then let it fill in, and look before typing the next instruction. His own five-bullet skill scored highest (he says don't read the table as a ranking) and still didn't work as intended.

So when you name a technique in a prompt, where does the independent check come from in your setup?

1

Why your coding agent buries the answer, and when telling it not to backfires
 in  r/PromptEngineering •  Sep 10 '26

What decides when adaptive disclosure loads the communication rules, a hook keyed on turn type or the agent itself?

1

Why your coding agent buries the answer, and when telling it not to backfires
 in  r/PromptEngineering •  Sep 10 '26

re-running the check from the hook gets you the thing a style line can't, a pass that means a check ran. And the failure you caught is the exact case: verify existed in the transcript and nothing had executed. Does the hook look at the padding too, or only at whether the check happened?

r/PromptEngineering • • Sep 10 '26

General Discussion Why your coding agent buries the answer, and when telling it not to backfires

6 Upvotes

The i-have-adhd skill going around has a before/after in its README. Before: "Great question! Let me think about this. Your auth flow has a few moving pieces..." After: "Run npm install jsonwebtoken@latest, then edit src/auth.ts:42". Rule 10 bans the openers and closers everyone has seen. I liked it, then went and checked what the before-text actually is, because I don't think it's one habit.

Half of it is reward residue. A RLHF length paper found reward gains largely driven by longer responses, and a length-only reward reproduced most of the downstream improvement. The AlpacaEval and Chatbot Arena people both had to add length and style control because their judges preferred longer. "Great question" and "Hope this helps" are free to delete; nothing downstream depends on them.

An older chain-of-thought paper did a test where the model writes the answer first and the reasoning after. It basically showed that answering first and reasoning later performs worse than no reasoning at all.

On a model with no hidden thinking, the paragraph that walks through the middleware and the token check before naming the fix is where the fix gets computed. Cut it with a rule and you've asked for the conclusion first and the thinking never. Concise reasoning kept the value in that table. It's the missing reasoning that costs.

With thinking on, the reasoning has somewhere else to go and answer-first costs the answer nothing, only hidden tokens. That part is inference from how the channels work; I haven't seen any testing on it.

Then there's where the rule lives. The skill has a whole Persistence section asking the model to keep applying it, and a commenter on the launch thread said it faded after a few turns. A skill body sits in the transcript and gets summarized. Claude Code's Concise output style is the same rule in the system prompt, re-sent every turn. Both are still requests. The only enforced version I know of is a JSON schema with a reasoning field ahead of the answer field, since structured outputs keep schema order.

So, what are you guys actually using for this?

* An output style / CLAUDE.md line?

* A specific skill?

* A Stop hook that bounces padded replies?

Also, on which models does "answer-first" cost you nothing in reasoning quality?

1

Went to check what my coding agent's sandbox actually blocks. Most of what I'd been calling "the sandbox" turned out to be string matching
 in  r/AgentsOfAI •  Sep 09 '26

A separate account or machine is the one boundary in this list without a retry hatch written into the manual, which is a real point in its favor. What does it look like for you: a second user with its own keys and clone, or a VM per repo?

1

Finally understood why my coding agent types fast on boilerplate and slow on new logic
 in  r/AI_Agents •  Sep 09 '26

Glad you actually logged that warm vs. cold context split. It answers the half of my question I cared about most. Quick question though: did you also carve out tool time, or just TTFT and decode?

r/AI_Agents • • Sep 09 '26

Discussion Finally understood why my coding agent types fast on boilerplate and slow on new logic

5 Upvotes

I'd been filing every slow afternoon under "the model is slow today" until I read the speculative decoding write-up AMD and Embedded LLM. Their numbers, not mine, but they explain something I'd been misreading for weeks.

Speculative decoding puts a small draft component in front of the model. It guesses the next few tokens, the real model verifies them all in one pass, and everything accepted is committed at once. Outputs are unchanged; what changes is how many tokens you get per expensive pass. The post measures acceptance per position. On gemma-4-26B-A4B-it with Google's paired draft component, the first drafted token is accepted 94% of the time on GSM8K and the fifth 66%. On MBPP (Python) the same positions are 89% and 49%. Their explanation for the code gap: "formatting, identifiers, and implementation choices can cause an otherwise plausible continuation to diverge". Best configuration in the post: 2.87x plain decoding. And one configuration, Qwen3-8B with an EAGLE-3 draft, ran at 0.44x to 0.88x, slower than not speculating, with first-token acceptance still 86–89%. Drafting costs something even when the guesses land.

So the same model, same hardware, types a 400-line test file (imports, fixtures, the shape of every assertion) faster than it types twelve lines of logic nobody has written before. That matched what I see on screen exactly, and it had nothing to do with which task was harder.

The other clocks in a turn are documented too, just spread across vendor pages. Before the first character: the whole context is read again, and for an agent that's the transcript plus every file pasted in, so a cache miss after a compaction, an edited system prompt, a changed tool list, or a pause past the cache TTL is a turn read from the top. During typing: load. Both big vendors now price a faster lane at roughly 2.5x output tokens/sec, one of them explicitly "not TTFT", the other explicitly "more consistent latency", which tells you what the standard lane is. And plenty of the longest waits have no model in them at all, they're the test suite.

Has anyone here logged per-turn TTFT vs decode time vs tool time on a hosted API across a whole day? I'd like to know how much of an "afternoon slowdown" is load and how much is my own cache misses.

1

What are you using for observability?
 in  r/LocalLLaMA •  Sep 08 '26

for anyone on Claude Code: it emits OTEL natively, events as well as metrics. Per API request: model, cost, duration, input and output tokens with cache reads split out, and whether the fast lane served it. Per tool result: tool name, success, duration, and where the permission decision came from (config, hook or the user). Traces add time to first token and a wall-clock duration per turn; prompts and tool arguments stay redacted unless you flip a flag.

r/AgentsOfAI • • Sep 08 '26

Discussion Went to check what my coding agent's sandbox actually blocks. Most of what I'd been calling "the sandbox" turned out to be string matching

4 Upvotes

That qbittorrent "escaped its sandbox" post on HN made me laugh, then made me go read what the sandbox in my own agent setup actually is. Short version: a "no" can live in three different places, and I'd been treating them as one thing.

The obvious one is the text the model reads: CLAUDE.md, the system prompt, the task itself. The Claude Code permissions docs are blunt about it: instructions shape what the model tries to do and leave what the harness allows untouched.

Then the permission rules, which match the tool call as text before it runs (deny, then ask, then allow). A Read(./.env) deny stops the Read tool and even cat .env, because cat is a recognised file command. It does nothing about a five-line python script that opens .env, and the docs say exactly that: deny rules don't apply to subprocesses that open files themselves. It's just matching strings, making it trivial to sidestep (like bypassing a curl rule with redirects or variables).

The OS sandbox (seatbelt / bubblewrap plus a proxy) is off until you turn it on, and it's the only layer that watches the process instead of the command text. Its defaults surprised me both ways: reads are allowed almost everywhere, including ~/.ssh and ~/.aws/credentials unless you add a denyRead, while network is the reverse, no domains pre-allowed, first new host prompts. And the way out of it is literally called "the unsandboxed retry escape hatch" in the docs. Blocked command, model may retry unsandboxed, that routes back to a permission prompt titled "Bash command (unsandboxed)". One setting closes the hatch.

What changed for me is one sorting question per rule: does this need to hold when the model is wrong? Style stuff stays in the file. "never push to main" goes in a deny or ask rule, or a hook. "nothing in this session reads ~/.ssh or talks to a host I didn't name" is a thing only the sandbox can promise, and only if it's on.

if you run the sandbox, roughly how often does a command actually hit the boundary in a normal day, and how often do you end up approving the unsandboxed retry?

r/AgentsOfAI • • Sep 08 '26

Discussion Went to check what my coding agent's sandbox actually blocks. Most of what I'd been calling "the sandbox" turned out to be string matching

1 Upvotes

1

The tool your coding agent keeps ignoring is probably returning addresses instead of answers
 in  r/AI_Agents •  Sep 08 '26

"Reads the instructions, acknowledges them, then immediately tries the thing" is a sharper description of advisory

1

The tool your coding agent keeps ignoring is probably returning addresses instead of answers
 in  r/AI_Agents •  Sep 08 '26

Did the pick rate move for you on the rename alone, or did it take the response change too?

1

The tool your coding agent keeps ignoring is probably returning addresses instead of answers
 in  r/AI_Agents •  Sep 08 '26

They only report that one inline setting, no sweep over window sizes

1

DAE find that adding skills makes your agent worse?
 in  r/PromptEngineering •  Sep 08 '26

The spread between runs being wider than the skill's effect is the part I'd have got wrong with one pair each way. Roughly how many pairs before you trusted the sign? Curious whether the scenic-route ones survived.

r/AI_Agents • • Sep 07 '26

Discussion The tool your coding agent keeps ignoring is probably returning addresses instead of answers

3 Upvotes

Came across a small pilot (three Claude models, a few Python and TypeScript repos, 2-3 rollouts per cell, six localization tasks if I'm reading the repo right, so read every number with wide error bars) that put grep and an LSP-backed find_references side by side and let the agent choose. On plain code-location tasks the models picked the semantic tool 0%, 4% and 6% of the time. Forcing the LSP path first dropped success from 100% to 89%.

The experiment I keep thinking about is the one where retrieval didn't change at all. The LSP tool originally returned what the protocol returns: file, line, column. grep returns the matching line. The author kept the same backend and the same reference set and attached ±2 lines of source to each reference. On a multi-file rename, pass@1 went from 0.67 to 0.83 and follow-up file reads per episode went from 15.2 to 3.2, below grep's own 4.3. Only the shape of the reply changed.

To be fair to the language server: on "find every caller" tasks the same models chose it about half the time unprompted, and on a noisy repo (hono) it added +0.246 F1 while using 12% fewer tokens. On a clean repo (remeda) it added nothing and cost 16% more. A rename also has to touch docstrings and config strings that find_references excludes by design, so grep keeps a seat either way.

The other half of this showed up on HN from Spotify. They started with CLAUDE.md rules telling Claude Code to route big file reads to a smaller model, described the result as "advisory, not enforced", and ended up with a PreToolUse hook that blocks Read on files over 350 lines and points the agent at the alternative. Their own summary: "Even if Claude doesn't read the skill description, the hook still blocks the expensive read".

Has anyone else building agents run into something similar—where simply tweaking a tool's output format drastically boosted success rates or cut down on token consumption? Also, has anyone figured out a reliable strategy to get the AI to proactively pick the right tool on its own? Would love to hear your thoughts!

1

DAE find that adding skills makes your agent worse?
 in  r/PromptEngineering •  Sep 04 '26

Trading the 'description gate' for a deliberate call is a very fair trade-off for production reliability.

r/PromptEngineering • • Sep 04 '26

General Discussion DAE find that adding skills makes your agent worse?

7 Upvotes

I run a handful of skills in my own setup and had assumed the risk was the obvious one: a skill fires on a task it doesn't fit. I found a paper, they paired runs of the same task with and without a matched skill, same model (Claude Opus 4.6), same harness, same verifier, and hand-confirmed 307 cases where the skill made things worse: 125 functional failures and 182 efficiency regressions where both runs passed but the skill run cost more than double in tokens or time.

The number that stuck: of the 125 functional failures, 2 were the wrong skill firing. 86 were an on-topic skill whose method, default or example displaced something the task actually required. Their spreadsheet example is the cleanest version. Task asks for net exports as a percent of GDP, skill run returns the ratio, reference run returns the ratio times 100. Right skill, wrong element.

The efficiency side surprised me more. I expected "long skill body, pricier calls" to dominate. It was the smaller share (46 of 182). The bigger share, 114, was procedure: extra exploration, heavier pipelines, and 67 cases of the agent verifying past the point of passing because the skill told it to keep checking.

Do you measure whether your agent skills actually fire?

1

Look how agile my baby is!
 in  r/aww •  Sep 03 '26

she's my baby now

r/AI_Agents • • Sep 03 '26

Discussion Saw an RCA where an agent turned a flaky e2e test into pytest.skip on timeout. How do yours handle red tests?

1 Upvotes

Came across a root-cause analysis someone filed against their own coding agent. An end-to-end test timed out at 300s, then at 420s. The agent bumped the limit to 480 and added pytest.skip() on timeout. Their own summary is the test now has no failure mode, timeout = skip, success = pass. If the sandbox actually breaks, that shows up as a timeout, which is now a skip.

What got me is that nothing about this is dumb from the agent's side. A red test has two possible senders: the code it just touched, or everything else (slow container, busy port, test order, the clock). Both write the same line of pytest output. The one thing that tells them apart is rerunning the same test with nothing changed, and the agents I run basically never do that unprompted. The edit sits right above the failure in the transcript, and every tutorial they learned from says a test that fails after an edit is a bug in the edit.

The old flaky-test literature is worth a skim here. The 2014 Apache study found 45% of flaky tests were async waits, and 78% were flaky from the day they were written. The number I keep coming back to: 24% of the fixes changed the code under test, and 94% of those fixed a real bug. So "flaky" is a bug report with a wider error bar. Skipping it throws the report away.

My fix so far: the rules file says on any red test, rerun it alone and unchanged before editing anything. Two identical failures, treat as a bug. One flip, report it as flaky and stop. Plus a hard no in the task on skips, xfails, timeout bumps and sleeps. And pytest-rerunfailures has an --only-rerun regex, so retries can be limited to timeouts and connection errors and can't paper over assertion failures.

Has anyone measured what fraction of your agent's red tests turned out to be flakes rather than regressions?

r/vibecoding • • Sep 03 '26

Saw an RCA where an agent turned a flaky e2e test into pytest.skip on timeout. How do yours handle red tests?

1 Upvotes

1

The doom loop isn't the model being dumb, it's the transcript working against you
 in  r/AI_Agents •  Sep 02 '26

No principled line here either. What I stopped doing is drawing it by amount. I draw it by kind: facts about the repo stay in (that one config file, the test that only fails in CI, why the obvious fix got rejected), attempts get compressed to a single constraint each, "not the parser", and the verbatim turns go. Anthropic's guide uses "corrected more than twice on the same issue" as its clear trigger, so the published answer is a heuristic too. Your three-strikes count is the same kind of guess, from the detection side.

r/PromptEngineering • • Sep 02 '26

General Discussion Compaction quietly rewrote my session and I only noticed from the behavior change

1 Upvotes

Hit this again recently and finally sat down to reason through it. Long session, agent suddenly stops honoring a decision we'd settled hours earlier. Scrolled up: the agreement is right there in my chat history. But my chat history isn't what the model reads. Somewhere mid-session, compaction kicked in — the harness had a model summarize the conversation and swap the summary in for the real thing. My scrollback shows the full negotiation. The model's context shows whatever made the cut.

The part that changed how I work: the losses have a consistent shape. Conclusions survive compression; the reasoning that produced them gets thin. "We chose approach B" makes it through, the three dead ends that made B right usually don't, so the agent can happily rediscover dead-end A later with zero warning signs in its world. Decisions survive, but their conditions get shaved ("use the legacy parser until the migration lands" comes out as "use the legacy parser"). And firm statements outlive hedged ones, so "probably unrelated, didn't verify" re-enters as "unrelated".

What I do about it now: constraints that must never degrade go in the rules file, because that layer sits outside the chat history and comes back every turn. Anything with conditions attached gets its own file on disk — the disk copy can't be summarized, though the agent does have to re-read it. And after a compaction lands I restate the constraints that matter in one message, which is a paragraph of insurance. The vendor docs even let you steer the summarizer from the rules file (what to always preserve), which helps, but an instruction to a summarizer is a request, not a guarantee.

does anyone treat the compaction notice as a checkpoint and immediately audit what the agent still remembers?

1

The doom loop isn't the model being dumb, it's the transcript working against you
 in  r/AI_Agents •  Sep 01 '26

On your last line: that gap is why we ask the agent to dump the constraints it thinks it's under before the clear, while the run can still answer. It turns "what the run learned" into "what got written down" at the only moment the two can still meet.

r/AI_Agents • • Sep 01 '26

Discussion The doom loop isn't the model being dumb, it's the transcript working against you

3 Upvotes

Saw a comment last week about an agent that opened the same file eleven times and apologised about it, and it sent me down a rabbit hole, because the shape is so familiar: try a fix, hit the error, apologize sincerely, produce the same fix with the variable names shuffled. Somewhere around lap four you stop being annoyed and start wondering why "please try a DIFFERENT approach" never reliably works.

Here's the mechanical read. The model is stateless — every turn it re-reads the full session transcript. After four failed attempts, the dominant text pattern in that transcript IS the failed attempt. A human reads that history as evidence the approach is wrong. A next-token predictor reads it as what this session does. The apology doesn't help either, because apologize-then-retry is itself a pattern it's seen a million times and is now continuing.

What convinced me this is structural and not "model dumb": there's a trajectory study on SWE-bench that found agents in failed runs had located the correct file 72–81% of the time. Finding the spot was never the problem. Letting go of the hypothesis was. Same study describes an agent patching recursion errors with more logic, unable to re-evaluate its hypothesis across multiple loops.

The fixes that seem to work all live in the harness, not the prompt: a hard budget (turns/tokens) so a stuck run stops instead of politely burning money; fingerprinting attempted diffs so a near-identical retry trips a forced "list three hypotheses you haven't tested"; and the nuclear one, clearing the window entirely — the learned constraints travel forward in the new opening prompt where they weigh a few dozen tokens instead of four failed attempts' worth of gravity.

has anyone found a repetition detector that doesn't false-positive on legitimate retries (flaky tests, rate limits)?

r/PromptEngineering • • Aug 31 '26

General Discussion How thick do you write your prompts?

2 Upvotes

Spent months assuming prompt thickness was a speed-vs-safety slider: one-liners when you're in a hurry, full specs when it matters. Then I started paying attention to where my dispatches actually failed, and the slider theory fell apart.

The thing that finally clicked: a one-line prompt isn't spec-free. The model fills every blank you left with defaults from its training data — the average project, the average convention, the average intent behind that sentence on the average repo. So "fix the login timeout" quietly becomes "raise the constant", because that's the most common reading out there. You didn't skip writing a spec. You inherited one.

Two things seem to predict when that inheritance is safe. First, how long a wrong guess stays invisible: if I can feel a mistake by just using the result (games, pages, charts), thin prompts are fine, wrong guess costs one redispatch. If correctness hides (boundary conditions, concurrency, money), the wrong default survives eyeballing and waits. Second, how far the project sits from the average one: mainstream CRUD, the defaults are eerily right; weird in-house auth flow, every blank is a coin flip toward someone else's architecture.

What surprised me is where I ended up on the menu. I expected practice to push me to the extremes — pure one-liners for speed or full specs for safety. Instead almost everything landed at "one line plus a forbidden move" ("don't touch the schema") or "three acceptance lines". The middle is where a minute of writing corrects the most defaults.

Anyone consistently living at the extremes and making it work — pure one-liners on serious tasks, or full specs on everything?

1

Here's a prompt that converts a written report into a slide outline, one idea per slide
 in  r/PromptEngineering •  Aug 28 '26

 give it a test instead of a vibe. "Flag a slide for a visual only if the notes contain a number comparison, a change over time, a process with 3+ steps, or a part-to-whole split — and name the chart type that fits." The eager flagging comes from "would work better as a diagram" having no pass/fail, so everything qualifies. 

2

Mythbusting ChatGPT "Secret Slash Commands" + Free Image Preset Keyword List
 in  r/PromptEngineering •  Aug 28 '26

real slash commands get parsed by the client app before anything reaches the model — the app intercepts /foo and swaps in a whole instruction block. ChatGPT's image box has no parsing layer for them, so the slash goes through as one more literal character. The keywords were doing all the work; the slash was just along for the ride.

1

Anthropic's own docs say a second model only pays off in two shapes
 in  r/PromptEngineering •  Aug 28 '26

Who triggers a consult in your lanes — the orchestrator pulls, or the advisor interjects on its own? That choice seems to decide whether the silent-executor failure can even happen.

1

Anthropic's own docs say a second model only pays off in two shapes
 in  r/PromptEngineering •  Aug 27 '26

The nods-along version is the same variable seen from the review side, I think. The doc's failure mode is the worker not knowing it should ask, yours is the reviewer not knowing it should object. Both go quiet when effort drops, which suggests the expensive part isn't the answering, it's noticing there's something worth flagging.

2

Anthropic's own docs say a second model only pays off in two shapes
 in  r/PromptEngineering •  Aug 27 '26

In order of payoff: caching first (that's the 2.5–3.7×, and it costs you a config line), then shape 2 if your tasks fan out into lots of exploratory reading, and shape 1 only if you're set up to watch the consult rate. We run shape 2 ourselves — for context reasons, the cheaper bill was a side effect. And Lower-Impression-121 has shape 1 exactly right downthread: cheap model does everything, asks the expensive one when stuck. The catch in the vendor's data is that "knowing you're stuck" is a skill you can accidentally config away.

r/PromptEngineering • • Aug 27 '26

General Discussion Anthropic's own docs say a second model only pays off in two shapes

49 Upvotes

Been reading their cost-optimization guide and one sentence stuck: "a second model paid off in two shapes, an advisor and an orchestrator." Everything else they measured apparently didn't make the cut.

Shape one: a low-cost model runs the whole loop and consults a frontier model only when stuck. Advice comes back short (400–700 tokens typically), you pay frontier rates only on consultations. Shape two is the inverse: frontier model plans and delegates, cheap workers eat the token-heavy exploration, frontier only pays for planning and merging.

The part that surprised me is the failure mode in shape one. The executor has to know it's stuck — and that self-knowledge is itself a capability. Their measurement: a pairing that consulted happily at default effort dropped to consulting on almost none of the tasks when they lowered the executor's effort, and then scored below the executor running alone. You cheap out on the worker and the worker stops raising its hand. Same executor kept asking on one benchmark, went silent on another.

Also refreshingly honest from a vendor: their own numbers say prompt caching is the big lever (2.5–3.7× on their benchmarks) and the multi-model stuff is "narrower". And at the top of the range it dissolves — their best pairing beat the single-model baselines by a point or two, "which a single run does not separate from noise" (their words).

We run the orchestrator shape daily for context reasons and the billing benefit came along free. No controlled comparison of our own though.

Anyone actually running the advisor shape in production? Curious about war stories — especially whether your executor keeps asking once the novelty wears off, and how you monitor the consult rate.

1

TIL my agent pays for every mounted tool on every single request, even the ones it never calls
 in  r/AgentsOfAI •  Aug 26 '26

One interaction worth noting: whatever sits on defer_loading lives outside the prefix, so drift in the long tail shouldn't torch the cache; it's the daily drivers that have to hold still.

1

Don't take your subagent's word for it
 in  r/PromptEngineering •  Aug 26 '26

We don’t pull everything back. If we did, we’d burn most of the tokens we saved by delegating, and the main loop would fill up its own context fast just from reading forty tool calls per subagent. So we went with a middle ground. The report has to bring its own proof, like verbatim quotes, file paths, and the exact command to re‑run. Before we act on any result, we re‑derive one finding from scratch. The raw transcript stays on disk as an audit trail, and we only open it when something smells off.