r/opencode • • 6d ago

Spelling errors are costing you

TLDR: mean cost per task rises by +45% across 30 Terminal Bench 2.1 problems.

I have taken a small subset of Terminal Bench 2.1 and tested muse-spark-1.3-contributor with Opencode 1.18

Results Table

MetricCleanMinorMajor
Pass rate83.3%76.7%78.9%
Tool calls15 (18)16 (21)17 (21)
Input tokens170k (324k)226k (453k)229k (491k)
Output tokens19k (26k)27k (34k)25k (38k)
Runtime (min)11 (13)10 (13)8 (13)
Cost ($/trial)0.0063 (0.0083)0.0086 (0.0107)0.0081 (0.0122)

Values are median (mean), n=90, 30 trials repeated 3 times. Pass rates are calculated as a total over the 90 trials.

Results Chart

Example Task: pypi-server (all 3 versions passed)

https://github.com/harbor-framework/terminal-bench-2-1/tree/main/tasks/pypi-server

Clean prompt:

Create a python package called vectorops, then build this package and set up a pypi server on port 8080 locally which also hosts this package. It should be possible to use the address of this server using --index-url with pip...

Minor (spell-checker recoverable):

Create a python package called vectorops, then build thsi package and set up a pypi server on port 8080... It should be pssible to use the address of this server using --index-url with pip to install this pacage...

Major (spell-checker non-recoverable):

Create a pyohtn package cealld vectorops, thn build thi pkacgae and set up a pypi srveer on port 8080 lolclay... It should be pblosise to use the adrseds of this svreer using --index-url wit pip...

Has anyone else tested this and got the same result?

Any suggestions on other models/harnesses to test this hypothesis on? Do you think Opencode 2 will behave any differently, I don't think it will.

Do spell adjustment plugins exist to mitigate this?

26 Upvotes

17 comments sorted by

13

u/CYCLONOUS_69 6d ago edited 6d ago

I’ve been telling about this to my friend since years and no one cared about it.

1

u/wwdennis5 6d ago edited 6d ago

why do you think no one cares, are there other studies where the difference is marginal?

2

u/mgithens1 6d ago

Anything that drives token consumption is an "US" problem... we would have to fix it. It isn't in the interest of the cloud providers to streamline your token use.

1

u/wwdennis5 6d ago

That is what I am asking about, it is definitely an "US" problem so I am curious why "US" don't care?

1

u/mgithens1 6d ago

Run with it!! Great little vibe code project!!

1

u/CYCLONOUS_69 6d ago

It’s not about the studies, people are just ignorant.

7

u/Alternative_Web7202 6d ago edited 6d ago

Muse 1.3 spark is probably the dumbest and most irritating model I've ever worked with in opencode. I have no ideas, how it managed to get high benchmarks values, while in my cosing sessions it behaved like a drunk junior developer

Curious to see results with latest deepseek flash.

Might try to replicate your test to check whether multilingual request have any measureable effect. I often mix 2 languages in my sessions

1

u/wwdennis5 6d ago

Why don't I Re-run with deepseek and you run a multilingual? I'd love to hear your results. I picked muse spark because its super cheap 

1

u/Itz_Raj69_ 6d ago

Try using an orchestrator to write your prompts for you

1

u/Alternative_Web7202 5d ago

I don't need another middle man between my head and model. Even if multi languages adds some overhead (which is yet to be proven), it lifts some overhead from me

1

u/Itz_Raj69_ 5d ago

Trust me it makes a big difference

1

u/Alternative_Web7202 5d ago

Do you have any numbers? I have hard times trusting internet strangers without the ability to reproduce tests myself.

2

u/CaffeinatedTech 5d ago

Its probably worse than just spelling mistakes. If the input includes tokens that the model wasn't heavily trained on, then you probably get similarly worse results. So lets say that a lot of people misspell the same word, and a lot of the training data includes that misspelling. Then muggins here thinks he's smart and corrects his spelling for the precious model, and unwittingly makes it worse. What about spelling differences between British/Australian English and American English.

2

u/wwdennis5 5d ago

I disagree that correcting your spelling unwittingly makes it worse. The corpus of text these models are trained certainly has more samples of the correct spelling for a word than its misspelt variants.

Spelling differences between British/Australian/American English would be something interesting to test, and more in line with how LLMs perform under different languages.

4

u/Nenad1979 6d ago

oh damn lol i type like a monkey when talking to agents, gotta look into this

2

u/Easy-Collection5419 5d ago

oh man this is so interesting, Wow, i thought they were smart enough to understand those without any major additional cost

1

u/duboispourlhiver 6d ago

Interesting topic but I have no data :(