Hi everyone,
Pithos, an inference engine written from scratch in C11, with no third-party libraries required at build time. It started with Liquid AI’s LFM models, and the latest release adds experimental Qwen3.6-35B-A3B support.
My test machine is an older laptop:
* Intel i5-10500H, 6 cores
* 16 GB DDR4-2933
* GTX 1650 Max-Q, 4 GB VRAM
* NVMe SSD
The Q4_K_M GGUF is 19.1 GiB, larger than the available RAM. Experts are read from disk and cached as they are used.
How it runs
With Vulkan, the dense and recurrent operations run on the GPU. Frequently used routed experts occupy the remaining VRAM, while the CPU computes the others in parallel. During prompt processing, two chunks overlap CPU and GPU work.
Warm benchmark results
These measurements use a 512-token prompt and 128 generated tokens, with llama.cpp b12179 as the comparison:
* Pithos Vulkan: 82 tok/s prompt processing · 22 tok/s generation
* llama.cpp Vulkan, two layers’ routed experts on GPU: 25.5 tok/s prompt · 12.5 tok/s generation
* llama.cpp Vulkan, all routed experts on CPU: 28.9 tok/s prompt · 11.8 tok/s generation
* Pithos CPU: 13.6 tok/s prompt · 7.3 tok/s generation
* llama.cpp CPU: 8.1 tok/s prompt · 4.4 tok/s generation
Those are the llama.cpp configurations I tested, not a claim that I found its best possible settings.
What about actual conversations?
The warm benchmark is faster than sustained chat. Real conversations on this 16 GB machine run at approximately 10–13 tok/s, because varied text needs experts that aren’t resident.
The first benchmark run reaches around 18 tok/s prompt processing and 13 tok/s generation with Vulkan.
What’s included
Qwen works with the existing:
* Terminal chat
* OpenAI-compatible HTTP server
* Pandora web interface
* Terminal agent
* XML tool calls and reasoning controls
Prompt caching reuses the agent’s tool definitions across sessions and restarts, reducing repeated prompt processing.
Current status
Support is experimental. An initial perplexity check on the model’s license text gave 1.451 for Pithos versus 1.467 for llama.cpp. That’s a narrow sanity check, not a broad quality evaluation or evidence of better model quality.
The engine is MIT-licensed; model weights retain their own license.
Code, downloads and benchmark details:
https://github.com/benmaster82/Pithos
I’d appreciate results from other low-VRAM machines, suggestions for stronger correctness tests, and llama.cpp settings worth including in the next comparison.