Chapter 18: Reinforcement Learning Fundamentals¶

Learn behaviour from interaction: MDPs, value functions, Bellman equations, temporal-difference learning, and tabular Q-learning on a gridworld you build yourself.

Metadata¶

Learning Objectives¶

• Formalize problems as MDPs — states, actions, transitions, rewards, discount
• Define value functions V(s) and Q(s, a) and the Bellman equations they satisfy
• Explain exploration vs exploitation and the epsilon-greedy strategy
• Implement temporal-difference learning and contrast it with Monte Carlo
• Implement tabular Q-learning and train an agent to optimality
• Tune discount, learning rate, and exploration schedules
• Recover the optimal policy from a learned Q-table
• Connect tabular methods to DQN and policy-gradient methods (REINFORCE)

What's Included¶

Notebooks¶

Scripts¶

• config.py — Chapter config: seeds, gridworld layout, hyperparameters
• gridworld.py — Deterministic gridworld MDP with step/reset and value iteration
• q_learning.py — Epsilon-greedy tabular Q-learning and policy extraction

Exercises¶

• Problem Set 1: MDPs & Bellman — Compute returns, run a Bellman backup, and read values off a gridworld
• Problem Set 2: Q-Learning — Train a Q-agent, extract its policy, and verify it reaches the goal optimally
• Solutions — in exercises/solutions/ (notebooks and solutions.py for CI)

Diagrams (Mermaid)¶

• rl_loop.mermaid, value_iteration.mermaid

Read Online¶

• 18.1 Introduction — Markov Decision Processes, value functions, and the Bellman equations
• 18.2 Intermediate — Epsilon-greedy exploration and model-free tabular Q-learning
• 18.3 Advanced — From tabular methods to REINFORCE, function approximation, and deep RL

Or try the code in the Playground.

How to Use This Chapter¶

git clone https://github.com/luigipascal/berta-chapters.git
cd berta-chapters/chapters/chapter-18-reinforcement-learning-fundamentals
pip install -r requirements.txt
jupyter notebook notebooks/01_mdps_and_values.ipynb

Created by Luigi Pascal Rondanini | Generated by Berta AI