Text generation from a large language model feels slow for a specific, structural reason: it is autoregressive. The model produces exactly one token per forward pass, feeds it back in, and runs the whole network again for the next token. Worse, each of those passes is memory-bound — the bottleneck is streaming the model's billions of weights out of GPU memory, not the arithmetic. So for a single sequence, your expensive accelerator spends most of its time waiting on memory while producing one token at a time. Speculative decoding is a clever way to break the one-token-per-pass rule — and remarkably, it does so without changing a single output token.
The core idea: draft, then verify
Speculative decoding, introduced by Leviathan et al. in 2023, pairs two models: the big target model you actually want to sample from, and a small, fast draft model that approximates it. The insight is that hard language-modeling tasks are full of easy sub-parts — obvious next words, boilerplate, common phrasings — that a much cheaper model can predict just as well.
The process runs in two phases. In the draft phase, the small model autoregressively guesses the next K tokens (say 4) — cheap, because it's small. In the verification phase, the big target model takes those K proposed tokens and scores them all in a single forward pass. This is the trick: a transformer can evaluate the probabilities for many positions in parallel just as easily as one, since verifying is not autoregressive. You then accept the longest prefix of the draft that the target agrees with, and correct the first disagreement.
# Illustrative: one speculative step
draft_tokens = draft_model.generate(context, k=4) # small model proposes 4
logits = target_model.forward(context + draft_tokens) # big model verifies ALL 4 in ONE pass
# Accept the longest matching prefix; resample the first rejected token from the target.
# Net result: often several tokens produced per single target-model pass.If the draft is good and 3 of the 4 tokens are accepted, you just produced roughly 3–4 tokens for the cost of one expensive target-model pass instead of four. That is where the speedup comes from.
Why it's a free lunch
The natural worry is that leaning on a weaker draft model must degrade quality. It does not — and this is the elegant part. Speculative decoding uses a carefully designed rejection-sampling step so that the accept/correct rule produces tokens drawn from exactly the target model's distribution. The original paper is explicit that it achieves faster "exact decoding" "without changing the distribution."
In other words, the output is mathematically identical to what the target model would have produced on its own (down to floating-point tie-breaking). The draft model only changes how many expensive forward passes it takes to get there — never what comes out. That is why speculative decoding is described as lossless: it is a pure latency optimization with no quality tradeoff to tune or regret. On T5-XXL, the paper reported a 2×–3× acceleration over the standard implementation, with identical results.
When it helps — and when it doesn't
Speculative decoding is not magic; its benefit scales with the draft's acceptance rate. Every rejected token is wasted draft work plus a target pass that advanced only one position — so if the draft rarely agrees with the target, you can even go slower than plain decoding. Two things drive good acceptance:
- A well-matched draft model. It needs to be cheap enough that its passes are nearly free relative to the target, yet aligned enough to guess the target's tokens often. A common choice is a small model from the same family as the large one.
- Predictable text. Content with lots of "easy" tokens (structured output, code boilerplate, common prose) gets high acceptance; highly surprising, high-entropy text gets less.
It also mostly helps latency for a single stream. On a server already saturating its GPU with large batches, the target model's forward passes are no longer memory-starved, so the spare-capacity argument weakens — which is why speculative decoding shines most for low-latency, low-batch serving. A family of variants reduces the setup cost by removing the separate draft model entirely — for example, having the model draft from its own earlier computation, or proposing tokens by looking up recent n-grams from the prompt — but the accept-and-verify core stays the same.
Key Takeaways
- LLM decoding is slow because it's autoregressive and memory-bound: one token per pass, each pass streaming all the weights.
- Speculative decoding has a small draft model propose K tokens and the target model verify them all in a single pass, often yielding several tokens per expensive pass.
- It is lossless — a rejection-sampling step guarantees the output matches the target model's distribution exactly (2×–3× reported on T5-XXL).
- The payoff depends on the draft's acceptance rate: use a cheap, well-aligned draft, and expect the biggest wins on low-latency, low-batch serving.
Squeezing latency out of LLM inference — speculative decoding, batching, quantization, and GPU cost — is one of the things we do at Teqners. If your inference is slower or pricier than it should be, we can help.