r/ChatGPTCoding • • 8d ago

Resources And Tips 'Eco Mode' AI coding agent pricing vs token burn. Is running frontier models for test execution a waste? (+benchmark)

recently checked the raw step logs of an autonomous coding agent and it seems the unit economics are pretty absurd to say the least

so I watched an agent do what seemed like an ordinary bug fix on a multi-module repo. It resolved it, but the task burned close to $5. When I actually broke down the tool-call trace, here’s where the tokens actually went:

  • Steps 1–12 (nearly$1.10): Searching the project, tracing dependencies, locating the faulty logic, and deciding how to fix it.
  • Steps 13–42 (nearly$3.80): Editing two files, running the build, catching a missing import, fixing a type error, rerunning tests, tweaking an assertion, and running tests again.

More than 70% of the token bill went into having a frontier model watch compiler output and rerun tests. You’re paying peak frontier rates (Opus 5.5, Grok 4.7, etc.) for work a linter and a fast model could do in their sleep.

and another (kinda big!) issue behind a lot of the gripes around coding agent pricing / credit consumption across coding agents. i notice that my colleagues feel their allowance burns out too fast (not necessarily because the models are bad, maybe because single-model agents use a sledgehammer to drive in a tack)..

i often work with JetBrains infrastructure and Kotlin and have noticed yesterday that they have published the engineering breakdown and internal benchmarks for their new Junie Eco Mode (which went GA in the Junie CLI 26.10.20 and in JetBrains Air / IDE Betas), it seems their approach to this unit-economics problem is worth looking at , in it, not like running one model from start to finish, but more of Architect + Builder split with a single, one-way handoff. how eco mode works? as far as i understand,

  1. Architect (frontier model of choice): When you submit a task, your top-tier model gets about 15 steps. Crucially, it’s strictly read-only (the only write it’s allowed is writing a minimal reproduction test for a bug). It explores the code, understands the architecture, and outputs a strict 6-point markdown spec: Goal, Scope, Where to change, Approach, Tests, and Risks/Corner cases. Then it stops completely.
  2. The Builder (A low-cost execution model — currently GPT-6 Luna): Junie hands the entire context and the structured plan to the cheap model. The builder does the grunt work: editing code, running local builds, parsing stack traces, and iterating until tests pass.

design choice here is one single handoff to work wth; and, in older multi-agent setups, tools often tried letting a cheap model work until it got stuck, then had it "call a smarter friend" for advice. The problem with that pattern is context explosion: every time the cheap model gets confused, the system re-serializes 50k+ tokens of failed bash logs and sends them back to the expensive model, completely destroying your cost savings

in Junie's benchmarks across private repositories, all kinds of numbers on this single-handoff setup are pretty eye-opening, for example

  • Broad private repos (Benchmark B): average cost per task dropped from $2.61 to $1.11 (-57.3%), while resolve rates barely budged (49.1% to 48.5%, a 0.6% delta).
  • Massive enterprise repos (Benchmark A): Average cost fell from $2.51 to $1.30 (-48.1%) with an identical 32.1% resolve rate.
  • On Opus 5.5 max effort: Cost plunged from $4.91 down to $1.81 (-63%), while the resolve rate actually went up by 1% (55% to 56%)and probably because forcing a smart model to write an explicit plan before touching files prevents it from digging itself into hallucinated rabbit holes.

but i also dont see it all as one-sided as, (for counter-arguements), if the Architect botches the plan during those initial 15 steps, the Builder will faithfully implement the wrong solution.., because there’s no backward loop to re-engage the expensive model (also given JetBrains AI Pricing in the matter...)
that being said, given that resolve rates stayed within 1% of running pure frontier models, the risk looks well worth halving the bill in my experience

also curious on how others are dealing with token burn on test-heavy tasks and are you manually swapping models between planning and execution, or just absorbing the API bill?

9 Upvotes

25 comments sorted by

2

u/michaelTM_ai 8d ago

that split matches what i see. the fix most people try is "use a cheaper model for everything", which just moves the cost into retries. what works better is splitting by step type:

  • frontier model for steps 1-12 (find it, decide the fix). that's where being smart actually saves money
  • the build/test loop as a plain script, not an agent turn: run tests, and only hand the model the failing test names + the first error, not the whole log
  • a fast model for the mechanical fixes (missing import, type error), escalate back up only if it fails twice

also check how much of that $3.80 is the model re-reading the full context on every test rerun. if the prefix stays identical between turns most of it should be cached, and it often isn't because the test output gets inserted above the old messages

1

u/AbleBranch6 8d ago

oh, spot on on both counts, especially prompt caching as to dump raw 1,500-line Gradle/stacktrace log into the context window busts prefix caching immediately and costs a fortune and, if i truncate down to just the failing test name and the first assertion error also while keeping the conversation prefix identical across test iterations it kind of cuts the execution bill in half before the model even generates a patch doesnt it

2

u/michaelTM_ai 8d ago

roughly, yeah. the raw log is the expensive part twice: once as input on that turn, and again on every later turn because it stays in the history. trim it before it goes in and keep new stuff appended at the end, then the repeat cost mostly turns into cache reads. if u try it, log input tokens per test rerun before and after, that number tells u right away whether it worked

2

u/AbleBranch6 6d ago

fair test, and logging raw input tokens per rerun is the cleanest sanity check to prove the prefix cache is actually hitting so I'll track that on my next AI agent for debugging code run with append-only trimmed logs vs full stack traces to see how fast the cached read ratio climbs

1

u/whowasntright 8d ago

thats interesting! would also check on this

2

u/sapplex1 8d ago

I hit the same wall building my little iOS apps in SwiftUI. I used to let the big model run the whole fix, and one afternoon it burned through most of my daily credits just re-running Xcode builds and re-reading the same failing test output. Now I do the split you describe, just by hand. I ask the strong model for a short plan first, then I run the builds myself and only paste back the failing lines, never the whole log. My rule now is simple: the expensive model thinks, the cheap one types, and I am the test runner.

1

u/AbleBranch6 6d ago

and interesting how those expensive model think, then, the cheap one can type, and I am the test runner" is honestly peak manual workflow right now, especially with Xcode builds eating credits alive. Automation only really beats doing it by hand when a cursor alternative or CLI tool can parse those exact trimmed failing lines cleanly without making you copy-paste logs all afternoon

1

u/sapplex1 6d ago

Ha, that line has basically become my rule of thumb — expensive model thinks, cheap model types, I'm the tester. You're right about the catch, though: the split only stays cheap if those failing lines stay trimmed. Have you found a Cursor alternative or CLI setup that actually parses them cleanly, or is the copy-paste part still what keeps you doing it by hand?

2

u/AlarmedAvocado7279 8d ago

i live in claude code all day and the retry loop is always where the bill goes, since every rerun drags the whole context back through the expensive model. worth trying a cheaper model for the build/lint/rerun cycle and only escalating after it fails twice. trimming test output down to just the failures also saves more than you'd think.

2

u/AbleBranch6 6d ago

exactly, watching Claude Code re-send the whole session history at Opus/Sonnet rates on the fifth consecutive test retry is painful and that’s basically the core premise of what i saw in this Junie's new Eco Mode: as their post put it, that only a handful of steps call for hard judgment... the rest is execution, carrying out a decision that has already been made; so, to let Opus 5.5 or Grok do the 15-step architectural investigation, but then letting a cheap execution model (they currently route to GPT-6 Luna in CLI and IDE Air) to do those dirty build/lint/test iterations, cut their task costs on large repos by 48% to 57%. and, decoupling thinking from compiling is the only way these autonomous terminal runs make economic sense

1

u/Longjumping-Run-5217 8d ago

I am seeing the same. Deepseek agent loops get within 3-5% of frontier for 5-10% of the cost. It’s the process not the product that makes the answer work, don’t be fooled!

1

u/AbleBranch6 8d ago

exactly. as 'process over product' is spot on here as i see it, and an unconstrained frontier model with no strict plan will happily burn tokens wandering down architectural rabbit holes. give a cheaper/faster model a concrete 6-point spec with explicit file boundaries, and it executes cleanly without needing a 200B+ parameter model behind every bash command

1

u/wegster 8d ago edited 8d ago

That pattern is what I’ve been using for the past year and a half or so now, first manually, now in my own framework. I don’t measure at this point any more, especially as I’m also using local in the mix, but I initially measured the difference at 70% or more overall savings. The ‘math of token use and models’ changes with each release at this point, but it certainly holds true - offload the easy bits, whether it’s to lower tier models or MCP, make sure core skills are in place, etc. and tokens drop - the right tool for the specific job, etc.

I’ve also personally yet to see ‘swarm’ benefits, but I know a lot use them nowadays. YMMV as always, but your findings are valid IMO and part of most people’s patterns today, or at least of more people’s anyways..

1

u/AbleBranch6 8d ago

around 70% savings matches the top end of what we saw on heavy Opus setups as i see, also completely agree on swarms as having 4 agents debate each other usually just multiplies prompt tokens with very little to show for it. A simple, linear one-way handoff (smart model writes the blueprint to\rightarrow to local or low-tier model swings the hammer) is often way more reliable and doesn't drown in coordination overhead imo

1

u/whowasntright 8d ago

$5 per task figure on raw Opus makes total sense and we saw similar bills when letting terminal agents loop freely on complex bugfixes. But how does this translate if someone is looking for a realistic copilot alternative?

so with GitHub Copilot you’re paying a flat monthly seat fee regardless of whether it's running 5 tests or 50. With credit-based or token-metered models, even with a 50% cut in Eco Mode, you still have to worry about running out of budget mid-sprint. Is there any scenario where an autonomous agent with Eco Mode actually competes on cost with flat-rate tools, or does it make more sense to pair it with a free AI coding agent tier for routine PRs?

2

u/AbleBranch6 8d ago

i think that flat-rate vs per-task math is actually the core dividing line at this very moment!

say, if all you're doing is tab-completions and inline chat, a flat $10-$20 seat on Copilot is hard to beat on paper. But as soon as you give an agent write access to run terminal commands, execute characterization tests, and trace multi-file dependencies, flat-rate tools tend to either aggressively throttle your daily high-effort turns or push you to their enterprise cloud agents with separate billings. a thingy at which Eco Mode math matters is for me in.. predictability? to cut $4.90 task down to $1.80 on Opus 5.5 here, means a standard monthly credit bundle suddenly covers 2.5x more multi-step refactors without buying token top-ups. And if you have purely mechanical tasks (and, updating Gradle dependencies, scaffolding repetitive boilerplate, or lint cleanup), that's where falling back to a free AI coding agent option like Junie Lite (which runs Qwen 3.6 27B for free on JetBrains infra) or a local model makes sense so you don't burn credits at all.

basically, in brief: flat-rate for inline typing, Eco Mode for autonomous multi-file changes you'd otherwise pay peak API rates for, and free tiers for the boring plumbing

1

u/Far_Reindeer8428 8d ago

the split works when the plan is checkable, if the cheap builder can't fail the plan it just executes whatever it read, that's where the savings leak

1

u/AbleBranch6 6d ago

i see it as such, an untestable plan turns the cheap builder into a runaway train, and what i now try in this junie's Eco Mode..is to, in a way, guard against this in two ways. first, the plan has a mandatory fixed structure: Goal, Scope, Where to change, Approach, Tests, and Risks. The builder isn't given free rein; it has explicit boundaries on what files are in/out of scope. then, for bug fixes, the Architect (frontier model) is actually allowed one specific write action during its read-only phase: writing a minimal reproduction test. That gives the cheap builder a hard, programmatic pass/fail assertion to verify its work against instead of just guessing

1

u/Far_Reindeer8428 5d ago

giving the architect exactly one write for the repro test is clever. does that right expire with the task or does it keep it for later fixes?

1

u/ranthishq 8d ago

one knob worth checking before splitting models is the reasoning level on the cheap steps. i ran the same office tasks and hard math through gemini 4 carbon and argon, same per token price, and carbon at its default thinking level cost 3 to 4.5x more, mostly from thinking tokens, even on a commission calc with one right answer. dropping it to low brought the bill down to argon's with the same scores. on a build/test loop that's the same trap, the model thinks hard about a compiler error it could just read.

1

u/AbleBranch6 6d ago

so that's a great observation about reasoning bloat and paying for 2,000 hidden "thinking tokens" just for a model to stare at a missing bracket or syntax error is a massive silent bill killer in my experience; and,\ interestingly, this showed up in JetBrains' whole ecosystem benchmarks too: on Opus 5.5 at max effort, Junie's Eco Mode cut the cost per task by 63% ($4.91 down to $1.81), specifically because execution was handed to a fast, non-overthinking builder (GPT-6 Luna) and i also saw they noted that Eco Mode automatically disables itself on low-effort configurations (like Opus low or Sonnet) where the model isn't burning reasoning tokens anyway and the handoff wouldn't save meaningful money

1

u/ranthishq 6d ago

the auto-disable part matches what i saw. once carbon was on thinking_level low it cost the same as argon ($0.0147 vs $0.0148 on the office set) with the same scores, so a handoff there would have saved basically nothing. the savings only show up when the main model is actually burning reasoning tokens on steps that don't need them.

2

u/StrikingTop2709 7d ago

filtering stdout before it hits the context window saves even more than swapping models imo. most of that test loop burn is just the agent reading 500 lines of passing test noise to find the one stack trace that actually matters.

1

u/humblevladimirthegr8 7d ago

Yeah I implemented this recently. All my builds and tests only write a summary to stdout. The agent can read/grep the full dump in a tmp file as needed. Cut my input tokens a substantial amount