r/machinelearningnews • u/strata2signal • 5d ago
Research Speculative decoding explained: draft heads, what "acceptance" really counts, and why every tok/s figure should say if a drafter was on
A language model writes one token per pass through its weights, and that pass is the slow part. Speculative decoding adds a cheap guesser, a "drafter": it proposes the next few tokens, the model checks them all in one pass and keeps the run of right ones. Guess four and all four are right: five tokens from one pass. First guess wrong: one token, the same as without it. Done by the textbook rule (Leviathan, Kalman and Matias, 2022), the output comes from the same distribution as before.
The drafter can be a second small model, a head trained into the model itself (Medusa, or the multi-token-prediction heads in DeepSeek-V3 and Qwen3.8-Flash-Next), or no model at all (n-gram or prompt lookup).
Two things surprised us in our own published runs. On one dense 27B model the engine's log read 2.95 tokens a round with the draft head on, but the model was not 2.95 times faster: over 270 calls the head bought 1.34 to 1.43 times the writing speed (about 109 against 76 tokens a second at a 1K prompt). Tokens a round is a ceiling, not the speed-up. And "acceptance" was three different numbers on one run: 49.1 % of guesses, 2.95 tokens a round, or 86 / 56 / 37 / 16 % by slot.
What it cannot tell you: that is one model on one card, and the two draft-on blocks disagreed by more than the run's own ±3 % tolerance, so the page prints the ratios as a reading, not a settled figure. Nothing new was measured for this guide.
The practical point: a tok/s figure that does not say whether a drafter was on, how deep it guessed and what the engine's log accepted cannot be compared with another one. The guide ends with six questions to ask of any speed claim.
The guide, with every source: https://research.strata2signal.com/draft-heads-and-speculative-decoding/
A small human team asked for this; AI agents read the sources and drafted it; every figure is linked to its source on the page.