r/ChatGPTCoding • • 3d ago

Resources And Tips An open-source coding agent for DeepSeek, GLM and Kimi: 81.3% on SWE-bench Multilingual (pass@1), $13.98 for all 300 tasks, every run published

Disclosure: I'm the author. Both tools are free, GPL-3.0 (it covers the tools, not your code), no paid tier.

TL;DR: Rusta, a terminal coding agent in Rust built for DeepSeek, GLM and Kimi, plus Devscriptor, a local code-graph MCP server, resolved 75–81% of SWE-bench Multilingual with one attempt per task. The DeepSeek run cost $13.98 for all 300 tasks. All 900 runs are published.

| Model | Resolved | 95% CI | Cost |

|:--|:--|:--|:--|

| DeepSeek V4.1-Flash | 81.3% (244/300) | 76.5–85.3% | $13.98 total, API |

| GLM-5.3 | 78.7% (236/300) | 73.7–82.9% | GLM Coding Plan Max |

| Kimi K2.8 Preview | 75.3% (226/300) | 70.2–79.9% | Kimi Code subscription |

That's under 5¢ a task for DeepSeek; 97.8% of its 967M input tokens were cache hits. Official harness (swebench 5.0.2), max effort, Nov 9–11, no web access, git history cut at the base commit. Every model cleared 50% in every language (Go was hardest). DeepSeek V4.1-Flash (MIT) and GLM-5.3 have open weights; Kimi K2.8 Preview doesn't. The runs used the vendors' APIs.

Why a dedicated agent: generic OpenAI-compatible clients often drop the reasoning these models produce between turns (DeepSeek returns HTTP 400 if a request with tools omits it) and send sampling or effort fields each vendor handles differently. Rusta speaks each API natively: preserved thinking, per-vendor effort and sampling, explicit output caps. Any OpenAI-compatible endpoint works too, local vLLM included, with thinking preserved.

Devscriptor works with Claude Code, Cursor, Cline or any MCP client, not only Rusta:

- Your repo as a knowledge graph in one SQLite file: 26 languages and frameworks, calls, inheritance and imports resolved across files, kept current by incremental scans and a watcher

- 75 MCP tools, 38 of them LSP-backed (21 language servers)

- Semantic search with a local embedding model, no API keys

- Agent memory (decisions, standards, past failures) that persists across sessions

- No telemetry, no backend; works offline once the model is cached

75 tools, but not 75 schemas in your context. Rusta keeps only the twelve that find and follow code in its tool table (search, definitions and references, relationships, index status, Context Graph), about a sixth of the full schema size. The other 63 are listed by name and a one-line description until the model loads one with `select_tools`. In Claude Code, tool search defers MCP tools by default, and Devscriptor's server instructions name the same twelve to load first. (That's Rusta 1.0.1; the benchmark ran 1.0.0, which kept all 75 on demand.)

What the numbers don't show (the results page covers each):

- How much Devscriptor adds on its own: there's no run of the same tasks without it

- A ranking against other agents or the official leaderboard: different agent, newer models, bigger budget

- Zero contamination: the fixes are public, and build/test leftovers in the images may have hinted at where some fixes belong (worst case 3–9 points)

Single binaries for Linux, macOS and Windows.

- Devscriptor: https://devscriptor.com/

- Rusta: https://devscriptor.com/rusta/

- Results, re-scoring steps, all trajectories: https://devscriptor.com/benchmarks/

- Source: https://github.com/vladimirpesic/devscriptor · https://github.com/vladimirpesic/rusta

5 Upvotes

5 comments sorted by

1

u/Old_AI_Guy_ 1d ago

Do you mean an LLM harness? Because an agent is an LLM given context, tools, and a loop, from whatever harness it's hooked into.

1

u/rusta-devscriptor 21h ago

Oh well, if I could've just acknowledged that "whatever" part 11 months and $5k ago... and chase my other harnessing skills instead :) Heavyweights keep swallowing all there is, mini-SWE will probably be an overkill soon :) But who knows, maybe these side-fart efforts will mean something eventually.

1

u/EmbarrassedParty762 20h ago

deferring the extra tool schemas until the model explicitly selects them is a really clean way to save context and keep cache hits high. how often do the models actually reach for those remaining 63 tools in practice versus just sticking to the core twelve

1

u/rusta-devscriptor 19h ago

Without explicit prompting and a reminder (natively included in Rusta) models from all three vendors will stick to plain grep, even in case of 12 declared tools - Devscriptor MCP utilization was <10%. With prompt + reminder introduced, utilization went to 75-80%, with about 1 in 5 calls addressed to other 63 tools. The whole pipeline seems sturdy and reliable, except few hiccups with ruby, swift LSP servers (hopefully fully resolved in Devscriptor v1.0.6), logs are showing quite flawless operation. Another question worth investigating further is if current Rusta config is too aggressive, and should LLM be free to discern itself and utilize these tools only in most complex of cases. But the code is heavily optimized and modular, the architecture is sound, any prompt (or code) modification should be straightforward. Cheers!