Tokens
The Mail Sorting Room: how AI reads your words
What are tokens?
AI does not read English like you do. It breaks text into small pieces called tokens: whole words, fragments, or single characters. Think of a mail sorting room where every package gets a tracking number before it ships downstream.
Teaching tokenizer
The default o200k_base mode uses a real offline BPE vocabulary and real token IDs. WordPiece mode uses the real greedy algorithm with a compact teaching vocabulary, so its IDs and [UNK] results are illustrative rather than BERT-compatible. Tokenization is model-specific—always use the tokenizer for the model you call.
Tokenizer family
Real offline BPE merges and vocabulary IDs used by modern OpenAI model families.
Byte-pair encoding · Production o200k_base vocabulary
Algorithm source24 characters7 tokens
Word / word-start
5
Continuation / byte fragment
0
Punctuation
2
Number
0
Conveyor belt
Hello#13225,#11␠how#1495␠does#2226␠AI#20837␠work#1101?#30
100
Hypothetical cost @ $3/1M
< $0.01
Tokens / chars
7 / 24
Compression
3.4 chars/token
Key Insight
More tokens means more cost, more context usage, and more work for the model. Pricing is usually per million tokens, not per word. Even the same input can have a different count under another vocabulary; an emoji may span multiple byte-level tokens. Next: those tokens fill a limited context window.
Check your understanding
1/5What is a token?
Sources, scope & review statusVerified Jul 29, 2026
Claims this lesson makes
- Stable conceptProduction tokenizers map text or byte fragments to vocabulary IDs; segmentation is tokenizer-specific.
Primary reading
Next content review: 2027-07-29. Source links establish the lesson's claims; interactive numbers remain labeled simulations unless explicitly identified as measured data.