Chapter 22: AI Safety & Alignment¶

Make models safe and fair, measurably: content filtering, reward modelling for RLHF, fairness metrics (demographic parity, disparate impact), and red-teaming.

Metadata¶

Learning Objectives¶

• Map the risk landscape — misuse, accidents, bias, and emergent failures
• Build a content filter and reason about precision/recall trade-offs in safety
• Score a reward model on preference pairs (the signal behind RLHF)
• Measure fairness with demographic parity difference and disparate impact ratio
• Red-team a model with adversarial prompts and a judge
• Understand alignment approaches: RLHF, constitutional methods, and their limits
• Reason about interpretability and why transparency aids safety
• Connect safety practice to governance and regulation (Chapter 25)

What's Included¶

Notebooks¶

Scripts¶

• config.py — Chapter config: banned patterns, fairness thresholds
• safety.py — Content filter, reward-model accuracy, red-team harness
• fairness.py — Demographic parity difference and disparate impact ratio

Exercises¶

• Problem Set 1: Filtering & Rewards — Block banned content and compute reward-model accuracy
• Problem Set 2: Fairness & Red-Team — Detect disparate impact and tally red-team failures
• Solutions — in exercises/solutions/ (notebooks and solutions.py for CI)

Diagrams (Mermaid)¶

• safety_pipeline.mermaid, rlhf.mermaid

Read Online¶

• 22.1 Introduction — The risk landscape and a measurable content filter
• 22.2 Intermediate — Preference data, reward-model accuracy, and the RLHF loop
• 22.3 Advanced — Demographic parity, disparate impact, and adversarial testing

Or try the code in the Playground.

How to Use This Chapter¶

git clone https://github.com/luigipascal/berta-chapters.git
cd berta-chapters/chapters/chapter-22-ai-safety-and-alignment
pip install -r requirements.txt
jupyter notebook notebooks/01_risk_and_filtering.ipynb

Created by Luigi Pascal Rondanini | Generated by Berta AI