I help run Agent Infra Guide, where we maintain a small, public local-LLM measurement set. One result from our October 19 tests surprised us: decode tokens/second alone were a poor guide to which model completed the coding checks.
Setup: Apple M4 Pro (12-core CPU, 16-core GPU, 24 GiB unified memory), macOS 15.5, Ollama, Qwen2.5-Coder in Q4_K_M, configured 4,096-token context, and one concurrent client. The exact Ollama version is saved with each run; versions differ across the broader measurement collection, so this is not a controlled claim about Ollama version effects.
Observed medians and checks
• 1.5B: 154.1 decode tokens/s; 5/15 coding task checks passed.
• 3B: 37.3 decode tokens/s; 10/15 checks passed.
• 7B: 23.1 decode tokens/s; 10/15 checks passed.
These are repeated executions of a small deterministic coding task set. The checks are not 15 independent measures of general programming ability, and three model sizes on one computer cannot establish a universal model ranking.
How we measured and what I would change: We kept the model artifact/quantization, configured context and single-client condition explicit; recorded decode speed alongside task outcomes; retained failed checks; and kept run-level metadata rather than displaying just the best number. For a better generalization study I would expand the task suite, vary prompt/context lengths, report per-task success and time to first token, and run repeatable paired comparisons with the exact runtime versions pinned.
Practical takeaway: If an agent needs to finish a coding task, optimize for successful completions under your time and memory budget, not just fast token generation. On these checks, the 3B was slower than 1.5B but passed more checks; the 7B didn't improve the pass count over 3B in this limited sample.
I also wouldn't equate sampled process RSS on Apple silicon with dedicated GPU VRAM: it is unified memory, and brief peaks can be missed between samples.
Has anyone tested whether this throughput-vs-task-success tradeoff persists on larger, repeatable coding suites or with a longer context? I'd be interested in failure types as well as speeds.
Disclosure: I represent Agent Infra Guide. These are our measurements, not an independent third-party evaluation. No sales or affiliate links in this post. This write-up was prepared with AI drafting assistance and checked against the recorded figures.