Every token an autoregressive LLM produces requires a full forward pass over billions of parameters, and at batch size one that pass is bound by memory bandwidth, not by arithmetic. The GPU spends most of its time streaming weights from HBM and almost none of it doing math. Speculative decoding exploits that slack: a cheap drafter proposes several tokens ahead, the big model verifies them all in a single forward pass, and every accepted token is one you got for free.
The result is 1.5x to 3x lower time-per-output-token on latency-critical paths, with no change to the output distribution when it is implemented correctly, and no retraining of the target model. This tutorial covers how it works, the four drafting strategies worth considering in 2026, how to turn it on in vLLM and in plain PyTorch, and how to measure whether it is actually helping you.
How it works
The loop is straightforward:
- A drafter proposes
kcandidate tokens (typicallyk = 3to5). - The target model runs one forward pass over the prompt plus all
kcandidates, producing the target distribution at each of thosek + 1positions. - A rejection-sampling test accepts the longest prefix of candidates consistent with the target distribution and resamples the first rejected token from a corrected distribution.
- You always keep at least one token per step, and at best
k + 1.
The key property, from the original Leviathan et al. and Chen et al. papers, is that step 3 is exact: the sequence of tokens produced is distributed identically to ordinary sampling from the target model. Speculative decoding is a systems optimization, not an approximation. If you see quality drift after enabling it, you have a bug or a mismatched tokenizer, not a tradeoff.
Two numbers govern the payoff:
- Acceptance rate (alpha): the fraction of proposed tokens the target accepts. Driven by how well the drafter mimics the target.
- Draft cost ratio (c): drafter latency divided by target latency per step.
Roughly, speedup is bounded by (1 - alpha^(k+1)) / ((1 - alpha) * (1 + c * k)). The practical reading: a fast drafter with a mediocre acceptance rate often beats a slow drafter with an excellent one, and pushing k too high wastes verification compute on tokens that will be rejected.
The four drafting strategies
| Strategy | Extra weights | Typical acceptance | Best for |
|---|---|---|---|
| Draft model (a small model of the same family) | Yes, a 0.5B-1B checkpoint | 0.6-0.8 | General chat, where a same-tokenizer small sibling exists |
| N-gram / prompt lookup | None | 0.3-0.6, spiky | Summarization, RAG, code edit, diff-style tasks with heavy copying |
| Medusa-style extra heads | Small trained heads | 0.5-0.7 | Fixed model you control and can post-train |
| EAGLE-3 (feature-level autoregressive drafter) | Small trained drafter | 0.75-0.9 | Highest-value latency-critical deployments |
Draft model is the easiest correct thing: pair a 70B target with a 1B model from the same family so the tokenizers match exactly. Mismatched tokenizers are the number one cause of broken speculative setups.
N-gram / prompt lookup deserves more attention than it gets. It has no model at all: it searches the prompt and generated text so far for the current suffix and proposes whatever followed it last time. On RAG answers that quote retrieved passages, or on "rewrite this file with the bug fixed" style code tasks, acceptance is high and the drafter cost is essentially zero. It is the first thing we try on document-heavy workloads.
EAGLE-3 is the current quality leader: it drafts in the target model's hidden-feature space rather than in token space, using a tree of candidates, and reaches acceptance rates high enough to sustain 2.5x-4x on chat workloads. The cost is training a drafter against your target model, and keeping it in sync when you change the target.
Turning it on in vLLM
The lowest-effort path in production. N-gram first, because it needs nothing:
vllm serve meta-llama/Llama-3.3-70B-Instruct \
--tensor-parallel-size 4 \
--max-model-len 8192 \
--speculative-config '{"method": "ngram",
"num_speculative_tokens": 4,
"prompt_lookup_max": 4,
"prompt_lookup_min": 2}'
With a draft model instead:
vllm serve meta-llama/Llama-3.3-70B-Instruct \
--tensor-parallel-size 4 \
--speculative-config '{"method": "draft_model",
"model": "meta-llama/Llama-3.2-1B-Instruct",
"num_speculative_tokens": 4}'
Or from the offline API, which is the fastest way to sweep k:
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Llama-3.3-70B-Instruct",
tensor_parallel_size=4,
speculative_config={
"method": "ngram",
"num_speculative_tokens": 4,
"prompt_lookup_max": 4,
},
)
out = llm.generate(prompts, SamplingParams(temperature=0.7, max_tokens=256))
Config keys move between vLLM releases; pin the version and read the flags for the exact tag you deploy. The concepts above are stable, the JSON spelling is not.
Doing it by hand in PyTorch
Worth writing once, because it makes the acceptance test concrete and it is the version you adapt when you have a custom model that no server supports.
import torch
@torch.inference_mode()
def speculative_step(target, draft, ids, k=4, temperature=0.7):
# 1. Draft k tokens autoregressively with the small model.
cand, dprobs, dfull = [], [], []
cur = ids
for _ in range(k):
logits = draft(cur).logits[:, -1, :] / temperature
p = torch.softmax(logits, dim=-1)
t = torch.multinomial(p, 1)
dfull.append(p)
dprobs.append(p.gather(-1, t))
cand.append(t)
cur = torch.cat([cur, t], dim=-1)
cand_t = torch.cat(cand, dim=-1) # [B, k]
# 2. One target forward pass over prompt + all k candidates.
full = torch.cat([ids, cand_t], dim=-1)
tlogits = target(full).logits[:, -(k + 1):, :] / temperature
tprobs = torch.softmax(tlogits, dim=-1) # [B, k+1, V]
# 3. Rejection sampling: accept the longest consistent prefix.
accepted = 0
for i in range(k):
q = dprobs[i].squeeze(-1) # draft prob
p = tprobs[:, i, :].gather(-1, cand[i]).squeeze(-1)
if torch.rand_like(p) < (p / q).clamp(max=1.0):
accepted += 1
else:
# resample from the corrected residual distribution
resid = (tprobs[:, i, :] - dfull[i]).clamp(min=0)
resid = resid / resid.sum(-1, keepdim=True)
fixed = torch.multinomial(resid, 1)
return torch.cat([ids, cand_t[:, :accepted], fixed], dim=-1), accepted
# all k accepted: take a free bonus token from the last position
bonus = torch.multinomial(tprobs[:, k, :], 1)
return torch.cat([ids, cand_t, bonus], dim=-1), accepted
Two notes for a real implementation. Keep separate KV caches for drafter and target and truncate both back to the accepted length after each step; getting this wrong produces subtly corrupted context that only shows up in long generations. And note that the resampling step draws from the renormalized residual max(0, p - q) over the full vocabulary, not from the raw target distribution: that correction is precisely what makes the output distribution exact.
Measuring it honestly
Turn it on, then prove it helped on your traffic:
- Acceptance length: mean accepted tokens per verification step. vLLM reports this in its metrics; below ~1.5 the drafter is not paying for itself.
- TPOT (time per output token) at your production concurrency, not at batch size one.
- Throughput (tokens/sec/GPU) across the whole server.
- Output equivalence: run a fixed prompt set with a fixed seed, speculation on and off, and diff. Any systematic difference is a bug.
The trap: speculative decoding trades compute for latency. At batch size one the GPU is idle enough that the extra verification work is free. Under heavy continuous batching the GPU is already compute-saturated, so speculation can reduce total throughput by 10-30% while improving per-request latency. On a busy shared endpoint that is a bad trade; on an interactive coding assistant or a voice agent with a strict first-token-and-flow budget, it is exactly the right one. Many teams end up with two pools: a speculative low-latency pool and a plain high-throughput batch pool.
A decision checklist
- Latency-bound, low-to-moderate concurrency, interactive users → try speculation.
- Offline batch scoring or high-QPS saturated serving → skip it, spend the effort on batching and quantization instead.
- Prompt-heavy workloads (RAG, code editing, summarization) → start with n-gram; it costs nothing.
- A small same-family sibling exists → draft model,
k = 4, measure. - Sustained high-value chat traffic and engineering budget → train an EAGLE-3 drafter.
- Always sweep
kin {2, 3, 4, 6}; the optimum is workload-specific and rarely the default.
Speculative decoding is one of the few optimizations that gives real latency wins without touching model quality. It is also easy to enable badly — mismatched tokenizers, unswept k, throughput quietly traded away on a saturated cluster. Measure acceptance length and TPOT under production concurrency before and after, and let those two numbers decide whether it stays on.