Skip to content
AI DemystifiedHow AI actually works
Lesson 20 of 22Advanced topics & capstoneintermediate5 min

Distillation

Teacher quality, student cost, and a confidence router

What is distillation?

Knowledge distillation trains a smaller student to mimic a larger teacher (soft labels or demonstrations). You trade some quality for much lower cost and latency. Production stacks often add a router cascade: try the student first; escalate hard or low-confidence cases to the teacher.

Lab honesty

Everything here is a fixed teaching fixture, not a training run. "After distillation" only rewrites student answers on easy and medium items so you can feel the quality gap shrink. Real distillation needs data, a training loop, evals, and calibration. Numbers are staged for teaching.

Serving mode

Student
50%
accuracy
Cost
8
Avg latency
80 ms
Teacher calls
0
Student calls
8
Teacher
100%
accuracy
Cost
96
Avg latency
420 ms
Teacher calls
8
Student calls
0
Router
75%
accuracy
Cost
32
Avg latency
185 ms
Teacher calls
2
Student calls
8

Portfolio tradeoff (baseline student)

Teacher quality lift
+50 pts vs student
Teacher cost multiple
12.0× student
Router vs teacher cost
33% of teacher spend

Confidence calibration lab

Accuracy asks whether an answer is right. Calibration asks whether “80% confident” is right about 80% of the time. A router depends on both.

student risk 33%
Brier ↓
0.213
ECE ↓
0.230
Coverage
75%
Retained accuracy
67%
20–40% · n=1
40–60% · n=1
60–80% · n=2
80–100% · n=4

At threshold 0.70, the student keeps 6/8 cases. The colored bar is mean confidence; the vertical marker is observed accuracy. This tiny fixture illustrates the metric, not a production calibration claim.

Inspect one sample

A bat and a ball cost $1.10. Bat costs $1 more than the ball. Ball cost?

Gold: $0.05

Student confidence: 88%

Router cascade · student · wrong

$0.10

cost 1 · 80 ms · student only

Miss: bad calibration sends hard items to the student, or the student simply lacks capacity. Raise threshold or improve distillation data.

Active mode rollup (Router cascade)

easy · studentok · c=1
easy · studentok · c=1
medium · studentok · c=1
medium · studentmiss · c=1
hard · studentmiss · c=1
hard · escalatedok · c=13
hard · escalatedok · c=13
medium · studentok · c=1

Why this matters for products

Frontier models set quality; economics often force a smaller model on the hot path. Distillation plus routing is how teams keep most of the quality without paying teacher prices on every easy request. Measure both accuracy and spend.

  • Failure modes: overconfident student (silent wrong answers), underconfident student (everything escalates), stale teacher labels after a model upgrade.
  • Product pairing: use cheap students for high-volume paths; reserve teacher/reasoning tiers for hard multi-step work (see Models routing advice).

See also

Cost and tier choice: AI Models. Hard multi-step compute: Reasoning Models. Faster same-quality serving: Speculative Decoding. Training pipeline context: How AI is Trained.

Key Insight

Distillation shrinks the quality gap at student prices. Cascades capture most of the teacher's accuracy when the student is calibrated: high confidence on easy wins, low confidence escalates the hard cases. Measure both accuracy and spend, not either alone.

Check your understanding

1/4

What is knowledge distillation?

Sources, scope & review statusVerified Jul 29, 2026

Claims this lesson makes

  • Stable conceptA confidence-routed cascade is only trustworthy when confidence calibration is measured, not assumed.

Next content review: 2027-07-29. Source links establish the lesson's claims; interactive numbers remain labeled simulations unless explicitly identified as measured data.