Chapter 24: Research & Cutting-Edge Techniques¶

Read and reproduce frontier work: implement scaled dot-product attention and a mixture-of-experts router from scratch in NumPy, and learn to turn papers into code.

Metadata¶

Learning Objectives¶

• Read a paper efficiently — abstract, figures, equations, then details
• Translate equations to code and verify against mathematical invariants
• Implement scaled dot-product attention and confirm its softmax properties
• Implement mixture-of-experts top-k gating and sparse expert combination
• Reason about scaling — why MoE buys capacity without proportional compute
• Sanity-check implementations with shape and probability assertions
• Survey current frontiers — long context, MoE, multimodality, reasoning
• Plan a reproduction of a paper end to end

What's Included¶

Notebooks¶

Scripts¶

• config.py — Chapter config: seeds, model dimensions
• attention.py — Numerically stable softmax and scaled dot-product attention
• moe.py — Top-k gating and sparse mixture-of-experts forward pass

Exercises¶

• Problem Set 1: Attention — Verify softmax invariants and attention output shape
• Problem Set 2: Mixture of Experts — Check top-k routing and the sparse forward pass
• Solutions — in exercises/solutions/ (notebooks and solutions.py for CI)

Diagrams (Mermaid)¶

• attention.mermaid, moe.mermaid

Read Online¶

• 24.1 Introduction — A method for reading papers and turning equations into verified code
• 24.2 Intermediate — Scaled dot-product attention, the core of the transformer
• 24.3 Advanced — Top-k gating and sparse expert combination

Or try the code in the Playground.

How to Use This Chapter¶

git clone https://github.com/luigipascal/berta-chapters.git
cd berta-chapters/chapters/chapter-24-research-and-cutting-edge-techniques
pip install -r requirements.txt
jupyter notebook notebooks/01_reading_papers.ipynb

Created by Luigi Pascal Rondanini | Generated by Berta AI