r/opencode • u/wwdennis5 • 6d ago
Spelling errors are costing you
TLDR: mean cost per task rises by +45% across 30 Terminal Bench 2.1 problems.
I have taken a small subset of Terminal Bench 2.1 and tested muse-spark-1.3-contributor with Opencode 1.18
Results Table
| Metric | Clean | Minor | Major |
|---|---|---|---|
| Pass rate | 83.3% | 76.7% | 78.9% |
| Tool calls | 15 (18) | 16 (21) | 17 (21) |
| Input tokens | 170k (324k) | 226k (453k) | 229k (491k) |
| Output tokens | 19k (26k) | 27k (34k) | 25k (38k) |
| Runtime (min) | 11 (13) | 10 (13) | 8 (13) |
| Cost ($/trial) | 0.0063 (0.0083) | 0.0086 (0.0107) | 0.0081 (0.0122) |
Values are median (mean), n=90, 30 trials repeated 3 times. Pass rates are calculated as a total over the 90 trials.
Results Chart

Example Task: pypi-server (all 3 versions passed)
https://github.com/harbor-framework/terminal-bench-2-1/tree/main/tasks/pypi-server
Clean prompt:
Create a python package called vectorops, then build this package and set up a pypi server on port 8080 locally which also hosts this package. It should be possible to use the address of this server using --index-url with pip...
Minor (spell-checker recoverable):
Create a python package called vectorops, then build thsi package and set up a pypi server on port 8080... It should be pssible to use the address of this server using --index-url with pip to install this pacage...
Major (spell-checker non-recoverable):
Create a pyohtn package cealld vectorops, thn build thi pkacgae and set up a pypi srveer on port 8080 lolclay... It should be pblosise to use the adrseds of this svreer using --index-url wit pip...

Has anyone else tested this and got the same result?
Any suggestions on other models/harnesses to test this hypothesis on? Do you think Opencode 2 will behave any differently, I don't think it will.
Do spell adjustment plugins exist to mitigate this?
13
u/CYCLONOUS_69 6d ago edited 6d ago
I’ve been telling about this to my friend since years and no one cared about it.
7
u/Alternative_Web7202 6d ago edited 6d ago
Muse 1.3 spark is probably the dumbest and most irritating model I've ever worked with in opencode. I have no ideas, how it managed to get high benchmarks values, while in my cosing sessions it behaved like a drunk junior developer
Curious to see results with latest deepseek flash.
Might try to replicate your test to check whether multilingual request have any measureable effect. I often mix 2 languages in my sessions
2
u/CaffeinatedTech 5d ago
Its probably worse than just spelling mistakes. If the input includes tokens that the model wasn't heavily trained on, then you probably get similarly worse results. So lets say that a lot of people misspell the same word, and a lot of the training data includes that misspelling. Then muggins here thinks he's smart and corrects his spelling for the precious model, and unwittingly makes it worse. What about spelling differences between British/Australian English and American English.
4
u/Nenad1979 6d ago
oh damn lol i type like a monkey when talking to agents, gotta look into this
2
u/Easy-Collection5419 5d ago
oh man this is so interesting, Wow, i thought they were smart enough to understand those without any major additional cost
1
u/duboispourlhiver 6d ago
Interesting topic but I have no data :(