Ch 22: AI Safety & Alignment - Intermediate¶
Track: Advanced | Try code in Playground | Back to chapter overview
Read online or run locally
You can read this content here on the web. To run the code interactively, either use the Playground or clone the repo and open chapters/chapter-22-ai-safety-and-alignment/notebooks/02_reward_models_and_rlhf.ipynb in Jupyter.
Chapter 22: AI Safety & Alignment — Notebook 02 (Reward Models & RLHF)¶
Alignment with human values often runs through RLHF: collect preferences, train a reward model, then optimize the policy against it. We measure the reward model's core competency.
What you'll learn¶
| Topic | Section |
|---|---|
| Preference data (chosen vs rejected) | §1 |
| Reward-model accuracy | §2 |
| The RLHF loop and its limits | §3 |
Time estimate: 3 hours
Key concepts¶
- Preference pair — humans pick the better of two responses.
- Reward model — learns to score responses to match human preference.
- Accuracy — how often the reward model agrees with held-out human choices.
- Reward hacking — the policy exploits flaws in an imperfect reward model.
RLHF turns human preferences into a reward model and optimizes against it. Reward-model accuracy is the measurable core, and reward hacking is the failure mode to watch.
Run the full notebook in the chapter folder for code and outputs.
Generated by Berta AI