Ch 18: Reinforcement Learning Fundamentals - Intermediate¶
Track: Advanced | Try code in Playground | Back to chapter overview
Read online or run locally
You can read this content here on the web. To run the code interactively, either use the Playground or clone the repo and open chapters/chapter-18-reinforcement-learning-fundamentals/notebooks/02_q_learning.ipynb in Jupyter.
Chapter 18: Reinforcement Learning — Notebook 02 (Temporal Difference & Q-Learning)¶
Without a model, the agent must learn from experience. Q-learning updates action-values toward a bootstrapped target after every step.
What you'll learn¶
| Topic | Section |
|---|---|
| Exploration vs exploitation (epsilon-greedy) | §1 |
| The TD target and the Q-learning update | §2 |
| Training to convergence | §3 |
| Extracting the optimal policy | §4 |
Time estimate: 4.5 hours
Key concepts¶
- Epsilon-greedy — explore with probability epsilon, exploit otherwise.
- TD target —
r + gamma * max_a' Q(s', a'), a one-step bootstrap. - Off-policy — Q-learning learns the greedy policy while behaving exploratorily.
- Convergence — with enough exploration and a decaying step size, Q converges to optimal.
Q-learning is model-free control: epsilon-greedy behaviour plus a bootstrapped TD update converges to the optimal action-values, from which the optimal policy falls out by greedy selection.
Run the full notebook in the chapter folder for code and outputs.
Generated by Berta AI