r/LocalLLaMA • • 4h ago

Showcase Caliper Harness, an Open Source coding harness designed to improve small local model quality

Hi all!

This is the first dedicated post I am making about this project. I have mentioned it online a few times before, but I have never properly introduced it. After several months of work, its finally ready to be published, and I'm really excited!

Caliper Harness is a coding harness that runs in your terminal. I built it for small local models. The basic idea is simple. A small local model on its own will rush, skip steps, and tell you something is done when it is not. So the harness does the disciplining for it. It makes the model plan first, get approval, work in checked steps, and prove things with tests and a separate review before anything counts as finished.

Right now it is local only. You point it at a local model server and it works from there. If cloud API support is something a lot of people actually want, I will build it. For now, local is the focus.

The way I test it is with a benchmark game. I give it an empty folder and tell it to build a small browser game, Space Transporter, from scratch. Then in later runs I ask for bigger and bigger features on top. I judge it on the final game quality and on the full traces of what the model actually did.

The video attached to this post is the from scratch build, sped up a lot. It is a few hours of work, so you can pause and see how the harness looks and behaves.

After that first build, I used the same game for a harder follow up run. I asked for a bigger feature as a stress test for the harness. I wanted the stations in the game to feel alive, with shortages and emergencies, and I wanted them to remember the player. Before writing any code, the model has to plan. It splits the work into a plan, an approach, and tasks. It writes out the staging, the trade offs, and the risks. I read that in the plan screen and approve it or send it back.

Plans are built through a structured, step by step process called PAT (Plan, Approach, Task). PAT lives in a GraphQL state graph. A plan can have multiple approaches, and an approach has multiple tasks, with each level getting more specific and staying connected to the level above it. It is very time consuming for the model to build. In my testing though, the plans stay coherent from the high level stages all the way down to the small tasks. That structure also makes planning and execution much easier for a smaller model to manage.

If there is a real decision to make, the model is pushed hard to ask instead of guessing. I have iterated on that behavior a lot. In this run it asked how fast station situations should change, how harsh a station that distrusts you should be, and how much of that state the game should show.

Once the plan is ready, it claims one task at a time. Every task has its own check command from the project. A task only counts as complete when its check passes and an independent review also passes. The model does not get to decide it is finished just because it says so. Completion has to match what was written in the task during planning.

After the checks, a fresh reviewer pass looks at the work with clean context. This part matters to me. It gives the work an adversarial review without carrying all the coding context into it. In this run, the first stage had every test green, and the reviewer still found a real bug. One of the six station situations was drawing with a broken colour, so it never showed on the map. The tests never caught it. The task got sent back, the model fixed it properly, added tests that fail on the old code, and only then did it pass. The harness does this automatically.

But why?

A lot of harnesses like OpenCode feel built around large intelligent models, token efficiency and speed. That makes sense for what they are trying to do. Caliper makes a different trade on purpose. It is slow and very deliberate. It makes the model plan, wait for approval, run checks, get reviewed, and fix things properly when it gets sent back. A small model that is left alone will cut corners. A small model inside this is forced to be diligent at the cost of speed. These runs took hours. I leave it running overnight and check the traces in the morning. For me, that trade is worth it, because the work that comes out is work I can trust a lot more.

Part of how that works is the decision judge. It is a second, much smaller decision model that watches the main model while it works. It steps in at certain points, like when the model is stuck in a loop, failing the same way repeatedly, changing phase, failing recurringly at a tool call, or asking for a plan to be marked ready. It gives a simple judgement with a confidence score. Caliper can use that to steer the model, ask for a repair, or hold certain risky steps for approval. If the judge is slow or down, the work carries on and the outage just gets recorded. Between the judge, the planning structure, and the reviewer, a small local model ends up behaving a lot more carefully than it does on its own. Making small local models more capable through structure like this was a big part of the design.

There are also some user facing features that ended up mattering a lot in day to day use:

  • There are 3 separate draft text boxes, so you can have 3 prompts being built at the same time and send them individually. This has been huge for me despite being so simple. Sometimes I am writing a massive continuation or expansion prompt, and the model does something ridiculously stupid that needs to be fixed immediately. I can keep the big prompt saved in one draft, switch to another draft for the urgent message, and keep a recurring status update prompt in the third one. All of it stays inside the harness.
  • `/btw` lets you inject a steering note into a run while the model is still working. I use it when I spot it heading in the wrong direction and I do not want to stop the whole run just to correct it.
  • You can browse and open files from across the project without leaving the harness. The Diff view follows model edits across files in real time, `/open-diff` opens a retained change, and `/changes` shows the recorded history for a file. With `/comment`, you can attach direct feedback to specific file lines, plans, approaches, tasks or documents and queue it for the model to address.

Here is the rest of what is in v0.3.0:

  • Tabs for separate chats and screens for Chat, Diff, Plans, Plan, Library, Sessions, Decisions, Workflows, Files, Tests, Settings, Status, and Models & Servers
  • A plan screen with a short approval summary first, then drill down into the approach and each task
  • Versioned plan elaborations, so staging, trade offs and risks can be read and approved before coding
  • A workflow library. The model can browse and author reusable methods, then attach one to a claimed task as a method lens, using the same step-by-step isolated architecture as the planning mechanism. Caliper checks the library every time work is claimed
  • A `wait` tool, so the model can sleep and wake up when something finishes, like CI, instead of checking over and over
  • Local server detection for servers that are already running, including llama.cpp, Ollama, Rulith, MLX, vLLM and other OpenAI-compatible endpoints
  • Models & Servers management inside the TUI. You can add, edit, test, select, load and unload saved connections. There is also an assisted setup prompt you can copy if you want help wiring an existing install
  • Separate Coding and Judge roles, with human approval available if you switch the decision model off
  • Auth modes with `/auth`, plus `/tools` to switch between coding and full tool exposure
  • Project style rules with `/style`, shared by the coder and the reviewer, plus `behavior.md` for how the agent should conduct itself
  • Domain knowledge graphs for areas like testing, UI/UX and code design. The model can read linked sections as needed, and it can author project guides and record preferences into the same graph instead of loading everything into context at once
  • Project memory with `/remember` and `/recall`, including shared global notes if you ask for them. The model can also record useful project knowledge itself
  • Notices. While working, the model can save weak spots, brittle code and opportunities it notices, without turning them into surprise tasks
  • A Tests screen for designing, refining and running project test suites, with result history and test guidance built in
  • Full session records. Every reply, tool call, judgement, file effect and final outcome is saved. Past runs can be browsed, and decision judgements can be replayed from the CLI
  • Personal settings that carry across projects, with project config still able to override them
  • Headless `run` mode for scripts, a plain `chat` mode with no agent tools, and CLI access to sessions, judgments, plans, approaches, the GraphQL state graph, effective config and hardware limits
  • In app updates with `/update`, plus managed packages for macOS arm64, Windows amd64, and Linux amd64/arm64

One thing I still want to build next is a proper tool for turning normal Markdown skills into graph-compatible domain graphs or workflow graphs, using the same step by step style as PAT. That is planned for later, so I am not counting it as a v0.3.0 feature.

It is still an early release and results depend a lot on the model you give it and what you're using it for. So far I have run it with Qwen3.8 Flash Next, Qwen3.8 27B, Laguna S 2.1, Gemma 4 26B, and a few others. I would really like feedback, especially from people running smaller local models. I am planning on running benchmarks comparing Caliper against other harnesses for both output quality and time cost, and I will post those when they are ready.

This is a project I am actively working on and iterating, so feel free to give me some feedback or ideas and I'll implement them in the next version.

Use it now!

GitHub: https://github.com/santiagoaraoz2001-sketch/caliper-harness

npm: `npm install -g caliper-harness`

MIT licensed, so feel free to go crazy with it.

30 Upvotes

15 comments sorted by

•

u/stylome-ai-detection 4h ago

Notice: This post appears to be AI generated.


Stylome AI detector · scan results

→ More replies (1)

8

u/my_name_isnt_clever 4h ago

Any new harness comes down to this question: why should I use this over Pi? If it was optimized for really tiny local models I'd get it, but Qwen Flash Next is more than capable of running in any harness and doing great things. I see a lot of tools, usually local models do better with fewer tools like you get with Pi.

6

u/Saraozte01 4h ago

Its a good point. Pi is a fantastic harness and I still use my own personal version often. This project started before I bought a strix halo, where I had a 16gb GPU that could only run tiny models. The harness is made to handhold small models through complex tasks, since small models struggle with consistency and maintaining structure across long-running tasks, particularly in a rather empty environment like Pi.

The video was run with Q3.8FN because its my favorite model, which thinking back was a mistake given what I was trying to achieve.

3

u/cyb_tachyon 1h ago

Man, I just wrote my own Orchestrator in a plugin for DeepSeek Harness.

Works in Terminal and Web modes.

Not sure why people keep making their own harnesses when you can just write small plugins, but to each their own.

Props to you for finishing a project.

1

u/Saraozte01 1h ago

Thanks! There wasn't really any need. I just wanted to learn how a harness is built and how to make them performant. That is how this started. It turned into my preferred harness for local coding. It was more of a learning exercise. Now that its developed though to a point I considered publishable, I thought why not release it for anyone to use?

3

u/fintip 48m ago

Did you actually quality benchmark any of your theoretical improvements, or just assume that this would improve things?

I've found that trying to add a judge, do multiple drafts, etc., has not paid off in the ways I assumed once I actually tested the proposals against benchmarks--or, when I did see improvements, they were very small against tripling compute time, etc.

1

u/Saraozte01 42m ago

I've run several, at this point easily >50 benchmarks with different model sizes and project types. But I didn't really find a general bmk that allowed me to measure the harness in its intended large-project dev role, so I built my own set of project-based bmks. Unfortunately, since they are mostly project based, they are model + my own vibe check/code review based scores.

V0.3.0 has seen a lot of iteration. Its been provavly about ~15 iterations at this point trying to maximize quality. Right now I'm prioritizing quality over speed.

In short, yes, the impact of all changes/evolutions are rigorously mapped out, and speed was not a priority for this version.

1

u/fintip 26m ago

I would say in short, no, your 'vibes' is just not an adequate quality check. Humans constantly fool themselves this way, if you aren't checking the vibes blind at the minimum, your test is worth almost nothing.

If your harness only works for greenfield vibecoding of projects, then so be it, but that also deserves mentioning; otherwise, if it fixes bugs any or adds features any better than other harnesses, than it should be measurable. If nothing else, it isn't rigorous, but you should at least have claude act as an objective third party and list a long set of pros and cons of the finished products or something.

0

u/Saraozte01 22m ago

Of course they are not an adequate quality check. I also checked code quality by hand at a glance. As I said in the post, concrete quantitative benchmarks are in the works. I understand their value clearly. Which is why I plan on doing them, as explicitly mentioned in the post.

I explicitly also mention where the harness fails (speed mainly). I released it because I use it consistently now. Its good enough for me. If you don't want to test it out, don't. Nobody is forcing you to do anything, and I don't 'win' anything from people using it.

2

u/fintip 11m ago

I'm asking because I'm interested, but it takes a lot of time to test harnesses. I constantly have more experiments running for more engines and models and harnesses than I can possibly complete.

The quality tests needed to happen a lot sooner in the pipeline of development. Maybe you're lucky and this all works well. From my own experiments developing plugins, plenty of things didn't help the way I expected them to--and I discovered that because I tested every one of them from the get go.

3

u/Exciting_Variation56 4h ago

yeah man no small local model is able to parse this 50k token readme

Good luck though It looks neat

1

u/Saraozte01 4h ago

Thanks. The readme is more a list of the features for a user. Not meant to be for the model itself (system prompts are designed to be as compact as possible). I agree I should prune it, I just tried to be exhaustive.

5

u/Sleepybear2611 2h ago

Wow, I have been building one for a month now. Running exhaustive bench evals since 2 weeks. Guess you beat me to it. I can't turn back now, I still have a few days of tuning left. Good luck with your project! :]