Ch 19: Model Optimization & Inference - Intermediate¶
Track: Advanced | Try code in Playground | Back to chapter overview
Read online or run locally
You can read this content here on the web. To run the code interactively, either use the Playground or clone the repo and open chapters/chapter-19-model-optimization-and-inference/notebooks/02_distillation.ipynb in Jupyter.
Chapter 19: Model Optimization — Notebook 02 (Knowledge Distillation)¶
Distillation trains a small student to mimic a large teacher, using the teacher's softened probabilities as a rich training signal.
What you'll learn¶
| Topic | Section |
|---|---|
| Soft targets vs hard labels | §1 |
| Temperature scaling | §2 |
| Why distillation works | §3 |
Time estimate: 2.5 hours
Key concepts¶
- Teacher/student — a big accurate model supervises a small fast one.
- Temperature — softens the teacher's distribution to expose 'dark knowledge'.
- Soft targets — carry inter-class similarity that hard labels discard.
Distillation transfers a teacher's soft probability structure to a smaller student via temperature scaling, often recovering most of the teacher's accuracy at a fraction of the cost.
Run the full notebook in the chapter folder for code and outputs.
Generated by Berta AI