Ch 24: Research & Cutting-Edge Techniques - Advanced¶
Track: Advanced | Try code in Playground | Back to chapter overview
Read online or run locally
You can read this content here on the web. To run the code interactively, either use the Playground or clone the repo and open chapters/chapter-24-research-and-cutting-edge-techniques/notebooks/03_mixture_of_experts.ipynb in Jupyter.
Chapter 24: Research & Frontier — Notebook 03 (Mixture of Experts)¶
Mixture-of-experts scales model capacity without scaling compute per token: a gate routes each token to its top-k experts. We implement the router and the sparse forward pass.
What you'll learn¶
| Topic | Section |
|---|---|
| The gating network | §1 |
| Top-k routing and renormalization | §2 |
| Sparse compute and why MoE scales | §3 |
Time estimate: 3 hours
Key concepts¶
- Gate — a small network producing a distribution over experts.
- Top-k — activate only the best k experts per token (sparsity).
- Renormalization — gate weights over the selected experts sum to one.
- Conditional compute — capacity grows with experts, cost grows with k.
Mixture-of-experts routes each token to its top-k experts via a gate and combines them with renormalized weights — decoupling model capacity from per-token compute, the trick behind many frontier LLMs.
Run the full notebook in the chapter folder for code and outputs.
Generated by Berta AI