Multimodal
How non-text inputs condition generation
More than words
Multimodal models can condition on more than text. An encoder turns an image, waveform, or other input into numerical features; the language model receives those features through an integration mechanism such as projected visual embeddings or cross-attention. Many systems still generate text autoregressively, but there is no single universal “multimodal token” architecture.
Lab honesty
Attachments are simulated scenes (icons + captions), not real vision or speech models. Token counts are teaching estimates so you can see non-text inputs consume finite compute and request capacity. Actual image/audio accounting and context limits are model- and provider-specific. Answers are scripted from sample metadata, not a live multimodal API.
Attach a sample image
an orange tabby cat sitting on a blue woven mat near a window
Simulated attachment · 64 image tokens
Shared context tray
text: 10 tokimage: 64 tok
74 / 256 toy units · illustrative shared budget
Prompt presets
Key Insight
Multimodal does not mean human perception. The system aligns patterns across representations and can still miss small details, spatial relationships, text in an image, or evidence that contradicts a likely caption. Evaluate each modality and the cross-modal task—not only whether the final prose sounds plausible.
Check your understanding
1/4What does multimodal mean for modern AI models?
Sources, scope & review statusVerified Jul 29, 2026
Claims this lesson makes
- Stable conceptMultimodal systems encode images, audio, or other inputs into representations that a language model can condition on.
Scope: Architectures differ: visual features may be projected into token-like embeddings or supplied through cross-attention. Provider token accounting is implementation-specific.
- Stable conceptConnecting visual and language representations does not guarantee human-like perception or faithful descriptions.
Scope: The tray uses simulated modality units and scripted metadata; it does not run a vision or audio model.
Primary reading
Next content review: 2027-07-29. Source links establish the lesson's claims; interactive numbers remain labeled simulations unless explicitly identified as measured data.