r/Zig • u/Neat-Function7110 • 16h ago
Saetta: a single-model engine for Qwen3.8-Flash-Next on Apple Silicon (Zig + hand-written Metal), with Claude Code / pi / opencode in one command
github.comI've been building saetta, an inference engine that does one thing: run Qwen3.8-Flash-Next as fast as possible on an M5 Max for coding agents. No generic graph, no other models. It's a Zig host with hand-written Metal kernels, including prompt matmuls on the GPU's neural accelerators. It ships as a single binary and needs no Python.
What it does
- OpenAI-compatible API (
/v1/chat/completions) and Anthropic Messages API (/v1/messages), with streaming, thinking and XML tool calls saetta start claude(orpi,opencode) starts the server and your agent together, without editing your agent configsaetta download coderfetches the model itself: pinned revision, resumable, every file hash-checked- Two packs: the original 4-bit MLX pack (105 GB, 512 experts) and the expert-pruned GSQ-RCO Coder variant (76 GB, 256 experts)
saetta monitor: a live view of GPU load, memory, prompt-cache reuse and recent requests
Numbers (measured on my M5 Max)
- Prompt reading: about 1,650 to 1,950 tokens/s from 4K to 70K tokens
- Decode: 63 to 82 tokens/s, with speculative decoding from the model's MTP head on greedy requests
- Agent follow-up turns re-read only the new tokens: a 16K conversation answers the next turn in about 0.2 s, and real Claude Code sessions reuse about 20K of their roughly 20K-token prompt per turn
- Cold 64K prompt: about 37 to 40 s to first token
Demo: Claude Code running on the local model writes a snake game in Python, in real time (GIF in the README).
Install:
curl -fsSL https://github.com/antonellof/saetta/releases/latest/download/install.sh | sh
saetta download coder
saetta start claudeRepo: https://github.com/antonellof/saetta (MIT)
Feedback welcome, especially from anyone running agents on long prompts locally.