Ch 24: Research & Cutting-Edge Techniques - Intermediate¶
Track: Advanced | Try code in Playground | Back to chapter overview
Read online or run locally
You can read this content here on the web. To run the code interactively, either use the Playground or clone the repo and open chapters/chapter-24-research-and-cutting-edge-techniques/notebooks/02_attention_from_scratch.ipynb in Jupyter.
Chapter 24: Research & Frontier — Notebook 02 (Attention From Scratch)¶
We reproduce scaled dot-product attention — the operation at the heart of every transformer — directly from its equation, then verify it.
What you'll learn¶
| Topic | Section |
|---|---|
| The attention equation | §1 |
| Why scale by sqrt(d_k) | §2 |
| Causal masking | §3 |
Time estimate: 3 hours
Key concepts¶
- Attention — a softmax-weighted average of values, keyed by query-key similarity.
- Scaling — dividing by sqrt(d_k) keeps softmax gradients healthy.
- Causal mask — prevents a position from attending to the future.
- Invariant — every attention row is a probability distribution.
Scaled dot-product attention is a masked, scaled softmax over query-key similarities applied to values. Reproducing it in NumPy — and asserting its invariants — demystifies the transformer.
Run the full notebook in the chapter folder for code and outputs.
Generated by Berta AI