Lesson 19 of 22Advanced topics & capstoneadvanced5 min

Speculative Decoding

Draft proposes, target verifies, stream accepts a prefix

What is speculative decoding?

Autoregressive decoding has a serial dependency: token t + 1 depends on the prefix through token t. Speculative decoding uses a cheaper draft model to propose several tokens (γ), then evaluates those positions with the target model in one verification pass. A greedy variant can keep a matching prefix; an exact sampling variant uses acceptance and correction rules instead of literal token equality.

Lab honesty

This page is a visual simulation with scripted draft mistakes, not real model weights or logits. Real speculative decoding samples from draft distributions and accepts with rules that preserve the target distribution in exact variants. Speedups depend on draft quality and cost, batching, hardware utilization, and γ; speculation can be slower when its overhead exceeds accepted work.

Demo prompt

Draft length γ4

Final stream

ofFranceisParis.Itisbeautiful.

Draft acceptedTarget bonusAcceptance 88% · speedup ~2.44×

Target forwards

vs 9 tokens out

Draft tokens proposed

γ = 4 per step

Est. speedup

2.44×

draft cost 0.15× target

Draft → verify steps

Exactness audit

Prefix matching above is the visual story. Production speculative sampling needs a correction rule at each position: accept proposed token x with min(1, p(x) / q(x)); after rejection, sample from normalized max(0, p − q). Here p and q are the target and draft distributions conditioned on the accepted prefix.

Reconstruction L1 error

5.6e-17

Token	Target p	Draft q	Accept chance	Residual	Final mass
Paris	0.35	0.55	64%	0.00	0.35
London	0.40	0.25	100%	0.75	0.40
Rome	0.20	0.15	100%	0.25	0.20
Berlin	0.05	0.05	100%	0.00	0.05

Rejection mass: 20%. The reconstructed final mass matches the target probabilities in this finite audit, which is the correctness contract the prefix animation alone cannot demonstrate.

Why this matters for products

Latency is a product feature. Exact speculative sampling is a serving-layer lever: it preserves the target distribution when implemented under the algorithm's assumptions, while seeking lower wall time from accepted draft work. That is different from "think longer" reasoning modes, which deliberately spend more inference computation to improve selected hard tasks.

Failure modes: weak drafts reject often (wasted work); γ too large with a bad draft hurts more than it helps.
Ops note: measure acceptance rate and p50/p95 latency, not just theoretical speedup.

Key Insight

Distribution preservation is a property of exact acceptance-and-correction algorithms, not of every draft-and-verify shortcut. A good draft can commit more tokens per expensive target pass; a weak or costly draft can waste work and erase the latency benefit.

Check your understanding

1/5

In speculative decoding, what does the draft model do?

Sources, scope & review statusVerified Jul 29, 2026

Claims this lesson makes

Stable conceptExact speculative sampling can preserve the target distribution while verifying multiple draft tokens in parallel.
Stable conceptFor sampling, correctness requires acceptance and residual-sampling rules; literal equality between a draft token and a target argmax is only a simplified greedy illustration.
Scope: The prefix animation illustrates a greedy mental model; the probability audit demonstrates the separate one-step exactness contract.
Stable conceptLatency gains depend on acceptance rate, draft cost, target hardware utilization, batching, and speculation depth, so speculation can fail to improve wall-clock latency.

Primary reading

Next content review: 2027-07-29. Source links establish the lesson's claims; interactive numbers remain labeled simulations unless explicitly identified as measured data.