Books and documentation
The chapter notes develop the module’s main material. These books, libraries, and tutorials provide alternative explanations, fuller proofs, and the tools used to run reinforcement learning experiments. The reading plan lists the lecture slides and readings for each topic, and cites the main references by the abbreviations in parentheses below or by the first author’s name.
Main references
| Source | Relevant material |
|---|---|
| Sutton and Barto (S&B), Reinforcement Learning: An Introduction, second edition (MIT Press, 2018) | Free, with a full PDF; the standard textbook and the backbone of Part I: chapter 2 on bandits (chapter 3), with section 2.9 on associative search (chapter 4); sections 3.1–3.8 on finite MDPs (chapter 1); chapter 4 on dynamic programming (chapter 2); chapter 5 on Monte Carlo methods (chapter 5), with sections 5.3–5.4 on control (chapter 7) and 5.5–5.9 on off-policy prediction (chapter 9); sections 6.1–6.3 and 6.8 on TD prediction and afterstates (chapter 6) and 6.4–6.7 on SARSA, Q-learning, and double learning (chapter 7); sections 7.1–7.2 and 12.1–12.8 on n-step returns and eligibility traces (chapter 8), and 7.3–7.6 and 12.9–12.11 on their off-policy forms (chapter 9); chapter 8 on planning and learning (chapter 10), 9–10 on on-policy approximation (chapter 11), 11 on off-policy approximation (chapter 12), 13 on policy gradients (chapter 13), 15 on neuroscience (chapter 6), and 16–17 on applications and frontiers (chapter 31); and section 17.3 on observations and state (chapters 1 and 14). |
| Murphy, Reinforcement Learning: An Overview (2024, updated 2025) | Free; a compact survey with consistent notation: sections 1.1–1.3 and 2.1 (chapter 1); chapter 2 on value-based RL, with 2.3 on learning values from samples (chapters 5–6) and 2.4–2.5 on SARSA and Q-learning (chapter 7), and as a whole for deep Q-learning (chapters 16 and 18); chapter 3 on policy-based RL: 3.1 on policy gradients (chapter 13), 3.2 on actor-critic methods (chapters 13 and 19), 3.3 on policy improvement methods (chapter 20), 3.4 on off-policy methods (chapter 9), and 3.6 on RL as inference (chapter 21); chapter 4 on model-based RL: 4.2 on decision-time planning (chapters 10 and 24), 4.3 on background planning (chapters 10 and 23), 4.4 on world models (chapter 23), and 4.5 on predictive representations (chapter 14); chapter 5 on multi-agent RL (chapter 27); sections 6.2 and 6.4 on RL for language models and its implementation (chapter 28); and sections 7.2 and 7.4 on exploration and intrinsic motivation (chapter 22), 7.3 on distributional RL (chapter 17), 7.5 on hierarchical RL (chapter 31), 7.6 on imitation learning (chapter 25), and 7.7 on offline RL (chapter 26). |
| Agarwal, Brantley, Jiang, Kakade, and Sun (RLTA), Reinforcement Learning: Theory and Algorithms (2026 draft) | Free; sample complexity, exploration, policy optimization, and imitation, with proofs, in the chapter numbering of the 2026 draft: chapter 1 (chapter 2), 3–4 (chapter 12), 4 (chapter 26), 5 (chapter 3), 6 (chapter 22), 11–12 (chapter 20), 13 (chapter 25), 14 (chapter 15), and chapters 1–13 together (chapter 30). |
| Lattimore and Szepesvári (LS), Bandit Algorithms (Cambridge, 2020) | Free; the reference on bandits: chapters 1–10 on stochastic bandits, concentration, explore-then-commit, UCB, and KL-UCB, 13–16 on lower bounds, 33 on pure exploration, and 36 on Thompson sampling (chapter 3); chapters 11–12 on Exp3, 18–20 on contextual and linear bandits and least-squares confidence bounds, 27–28 on adversarial linear bandits and mirror descent, and 34–36 on Bayesian bandits, the Gittins index, and Thompson sampling (chapter 4). |
| Bertsekas, A Course in Reinforcement Learning, second edition (2025) | Free; dynamic programming, rollout, and approximation from the control viewpoint: chapter 1, an overview of exact and approximate dynamic programming that includes linear–quadratic problems and AlphaZero’s off-line training and on-line play (chapters 2, 14, 15, and 24), and chapter 2 on approximation in value space with lookahead, rollout, and tree search (chapters 10 and 24). |
| Kochenderfer, Wheeler, and Wray, Algorithms for Decision Making (MIT Press, 2022) | Free, with code: chapter 7 on exact solution methods (chapter 1), 8 on approximate value functions (chapter 11), 9 on online planning and 16 on model-based methods (chapter 10), 10–13 on policy search, policy gradients, and actor-critic methods (chapter 13), 17 on model-free methods (chapter 5), 18 on imitation learning (chapter 25), 19–23 on beliefs, exact, offline, and online belief-state planning, and controller abstractions (chapter 14), and 24–27 on multiagent reasoning (chapter 27). |
| Szepesvári, Algorithms for Reinforcement Learning (Morgan & Claypool, 2010) | Free; a concise, rigorous treatment: chapter 1 and appendix A on MDPs and the theory of discounted MDPs (chapter 2), section 2.1 on temporal-difference learning in finite state spaces (chapter 6), and section 2.2 on value prediction in large state spaces, including gradient-TD methods (chapters 11–12). |
| Bellemare, Dabney, and Rowland (BDR), Distributional Reinforcement Learning (MIT Press, 2023) | Free; the reference on return distributions: chapters 1–10, from the distribution of returns, operators and metrics, and distributional dynamic programming to incremental algorithms, control, statistical functionals, linear function approximation, and deep distributional agents (chapter 17). |
| Albrecht, Christianos, and Schäfer (MARL), Multi-Agent Reinforcement Learning: Foundations and Modern Approaches (MIT Press, 2024) | Free; chapters 3–10 on games, solution concepts, foundational algorithms, and multi-agent deep RL (chapter 27), with lecture slides and a lecture playlist. |
| Tedrake, Underactuated Robotics (MIT course notes) | Free; dynamic programming, LQR, trajectory optimization, and policy search for robots: chapter 8 on linear quadratic regulators and 10 on trajectory optimization (chapter 15), and 21 on imitation learning (chapter 25). |
Libraries and environments
| Source | Relevant material |
|---|---|
| Gymnasium | The standard environment interface, which reports terminated and truncated separately (chapters 1 and 7) and runs vector environments with their reset conventions (chapter 19; Lab 10). Its toy-text environments expose their models in env.unwrapped.P: FrozenLake, Taxi, Blackjack, and CliffWalking (chapters 2, 5, 7, and 10; Labs 1, 3, and 4). Its classic-control tasks include MountainCar, CartPole, and Pendulum (chapters 11–12, 15–16, and 23; Labs 5–7, 11–12, and 14). The Box2D and MuJoCo extras, installed with pip install "gymnasium[box2d,mujoco]", add LunarLander and the MuJoCo locomotion tasks such as HalfCheetah (chapter 21; Labs 10–11). |
| MinAtar | Miniature versions of five Atari games on a \(10\times10\) grid (Young and Tian, 2019), which make deep RL experiments feasible without a GPU; installed with pip install minatar (chapters 16 and 18; Lab 8 and an extension of Lab 9). |
| Bellemare et al., The Arcade Learning Environment (2013) | The Atari benchmark of deep RL, with the sticky actions and evaluation protocols of Machado et al. (2018) (chapters 16–18). |
| Tassa et al., DeepMind Control Suite (2018) | Continuous-control tasks with bounded rewards and proprioceptive or pixel observations, on which many results of chapters 21 and 23 are reported. |
| Batched simulators: EnvPool (Weng et al., 2022), Isaac Gym (Makoviychuk et al., 2021), and Brax (Freeman et al., 2021) | Thousands of environments stepped in parallel, in a C++ thread pool or on the GPU, the last with differentiable physics written in JAX (chapter 19). |
| CleanRL | Single-file, benchmarked implementations of DQN, C51, Rainbow, PQN, PPO, DDPG, TD3, SAC, and RND, with documentation pages such as those for DQN, C51, and PQN; the PPO page documents its variants line by line (chapters 16–18 and 20–22). Chapter 20 also compares the PPO defaults of Spinning Up and Stable-Baselines3. |
| PureJaxRL | Training loops compiled end to end in JAX, environments included, and vectorized across random seeds on one accelerator (chapter 19). |
| mctx | DeepMind’s JAX library for AlphaZero, MuZero, and Gumbel MuZero search (chapter 24); Lab 13 follows its schedule for sequential halving. |
| Minari and CORL (Tarasov et al.) | Minari hosts the standard offline datasets, including those of D4RL (Fu et al., 2020), and CORL gives single-file implementations of offline RL algorithms; OGBench (Park et al., 2025) adds goal-conditioned tasks (chapter 26). |
| PettingZoo, OpenSpiel, and JaxMARL | Multi-agent environments and algorithms: PettingZoo’s multi-agent version of the Gymnasium interface, OpenSpiel’s games and game-theoretic algorithms, and JaxMARL’s common environments reimplemented in JAX to run thousands in parallel (chapter 27). |
| TRL’s GRPOTrainer and verl | Implementations of GRPO and the other policy-gradient algorithms for training language models (chapter 28). |
| Open Bandit Pipeline | Logged bandit feedback with propensities from a fashion e-commerce platform, for testing off-policy evaluation (chapter 4). |
Tutorials and hands-on material
| Source | Relevant material |
|---|---|
| OpenAI Spinning Up: introduction to policy optimization and the pages on DDPG, TD3, and SAC | A step-by-step derivation of the policy-gradient estimators (chapter 13), and concise accounts with pseudocode of the three off-policy actor-critics (chapter 21). |
| Hugging Face Deep RL course | Notebooks for Q-learning, DQN, REINFORCE, A2C, PPO, and self-play: unit 2 on Q-learning and Monte Carlo versus TD learning (chapters 6–7), unit 3 on deep Q-learning with Atari games (chapter 16), unit 6 on actor-critic methods (chapter 19), unit 7 on multi-agents and AI versus AI (chapter 27), unit 8 on proximal policy optimization (chapter 20), and bonus unit 3 on model-based RL (chapter 23). |
| Hugging Face LLM course, chapter 12: Open R1 and GRPO | GRPO and Open R1, an open reproduction of reasoning training with reinforcement learning (chapter 28). |
| ShangtongZhang/reinforcement-learning-an-introduction | Sutton and Barto’s figures reproduced in Python (chapters 1–3 and 5–13). |
| CS285 homeworks | Imitation learning (chapter 25), policy gradients (chapter 13), Q-learning and actor-critic algorithms (chapters 16 and 21), model-based RL (chapter 23), and exploration and offline RL (chapters 22 and 26). |
| David Silver, Easy21 assignment | Monte Carlo control, SARSA(λ), and linear function approximation on a card game like blackjack (chapters 5, 8, and 11). |
| Gymnasium’s vectorized A2C tutorial | Training an advantage actor-critic on parallel environments (chapter 19). |
| Lilian Weng, Policy gradient algorithms (2018, updated) and Exploration strategies in deep reinforcement learning (2020) | Surveys with consistent notation: policy-gradient algorithms from REINFORCE and actor-critic to A3C, IMPALA, TRPO, PPO, DDPG, TD3, and SAC (chapters 13 and 19–21); count-based bonuses, curiosity, random network distillation, episodic memory, and Go-Explore (chapter 22). |
| Huang et al., The 37 implementation details of proximal policy optimization (2022) | The code-level choices of the original PPO implementation, reimplemented one at a time (chapter 20); Lab 10 removes several of them to measure their effect. |
| Julian Schrittwieser, MuZero intuition (2020) | MuZero’s learned model and search, explained by one of its authors (chapter 24). |
| Kumar and Levine, Offline reinforcement learning: from algorithms to practical challenges (NeurIPS 2020 tutorial) | A tutorial on offline RL algorithms and the practical challenges of using them, to read with the review of Levine et al. (2020) (chapter 26). |
| Nathan Lambert, Reinforcement Learning from Human Feedback | A free book on post-training with a course; its chapter 6 and lectures 3–5 are the readings for chapter 28. |