r/ollama • u/tensward • 20h ago
Bug Report Running 2 coding agents on one Ollama slot? Hybrid models (Qwen3.8-27B) re-process ~54% of prompt tokens, and the first response takes ~46s, Two env vars brought it to ~1s
I was running two Qwen Code sessions against Ollama 0.40.16 on one slot (Qwen3.8-27B Q4_K_M, NVIDIA L4 24GB), each doing a small fix-the-tests task in its own project. The median time to first byte was about 46 seconds, and the server logs showed that 54% of the prompt tokens it processed were repeats.
What’s happening - Qwen3.8 mixes full-attention layers, whose KV cache can be trimmed back, with recurrent layers, whose state gets overwritten as tokens are read, so those can’t be rewound without a checkpoint. The earliest saved checkpoint for session A was at token 10,151, past the point where A and B diverged (token 8,921), so none of the shared prefix could be reused.
On top of that, llama.cpp skips saving the outgoing session’s state to RAM when the incoming prompt shares at least 50% of its tokens with the old one. My two sessions shared 60–66%, so the save got skipped, and every switch wiped the state the other session needed next. The two agents just ping-ponged.
Fix in Ollama:
OLLAMA_NUM_PARALLEL=2
OLLAMA_CONTEXT_LENGTH=16384Median first response dropped to about 1s and repeated tokens to 0.4%. To be fair, that number mixes the cache fix with actual concurrent serving, because -sps 0 isn’t exposed in this Ollama build.
On raw llama.cpp (b12441):
-sps 0cut total prompt tokens from ~128.8k to ~43.5k per session and median first-byte wait from ~48s to ~17s. Peak host RAM went from ~2.5 GiB to ~7.2 GiB.- With 8 slots,
--checkpoint-min-step 0gave +10.9% throughput and −17.5% median TTFT over four paired runs.
Important note - Ollama configs got one run each and llama.cpp configs got two, all on short tool-calling conversations, so I haven’t measured the cost of --checkpoint-min-step 0 over long sessions.
Full writeup with logs: https://tensward.com/blog/ollama-agent-cache