Reading plan

Work in progress: this part of the notes is still being revised.

Courses

The module has two parts. Part I develops classical reinforcement learning in depth, from Markov decision processes to policy gradients, partial observability, and optimal control: the theory and algorithms that every later method builds on, with tabular and linear methods that can be analyzed exactly. Part II develops modern reinforcement learning with deep networks and follows it to the current frontier: deep value-based and actor-critic agents, exploration, world models, planning with learned models, imitation and offline RL, multi-agent RL, reinforcement learning for language models, generalist agents, the modern theory, and deployment. Each block lists the material to cover and its primary lectures; further reading and the labs provide additional depth. Timing is flexible; Part II assumes Part I.

Video links point to the lectures on YouTube where the individual video has been identified, and otherwise to the course playlist with the lecture’s position. Several prerequisites come from earlier modules: expected utility and decision networks from AI chapter 12, game trees, minimax, and Monte Carlo tree search from AI chapter 4, equilibria and no-regret learning from AI chapter 15, filtering in hidden Markov models from AI chapter 11, stochastic gradient methods from DL chapter 3, and the score-function estimator from Generative AI chapter 3. Reinforcement learning from human feedback and for reasoning in language models is introduced in NLP chapters 11 and 12; block 28 develops its algorithms without repeating that material.

Topic list

Part I — Classical reinforcement learning

Part II — Modern reinforcement learning

Reference books used below

Part I — Classical reinforcement learning. Blocks 1–15 follow Sutton and Barto, Silver, and the DeepMind x UCL lectures, with bandit theory, partial observability, and optimal control added from the supplements.

1. Markov decision processes {#1-markov-decision-processes}

Topics: Agents, environments, rewards, and the reward hypothesis; what distinguishes reinforcement learning from supervised learning; Markov processes, Markov reward processes, and Markov decision processes; returns, discounting, and episodes; policies, state values, and action values; the Bellman expectation equations; discounted occupancy measures; optimal values, the Bellman optimality equations, and the existence of deterministic optimal policies; finite and infinite horizons; partial observability and belief states, previewed; other formulations.

Further reading: Bellman, Dynamic Programming (Princeton University Press, 1957); Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming (Wiley, 1994), the standard reference on the theory; Blackwell, Discrete dynamic programming (1962); Silver et al., Reward is enough (2021); Abel et al., On the expressivity of Markov reward (2021); Bowling et al., Settling the reward hypothesis (2023).

2. Dynamic programming {#2-dynamic-programming}

Topics: Policy evaluation by iteration and by solving linear equations; the policy improvement theorem; policy iteration; value iteration; contraction mappings and convergence rates; modified policy iteration; asynchronous, Gauss–Seidel, and prioritized updates; generalized policy iteration; the linear-programming formulation and its dual; the complexity of solving MDPs; the curse of dimensionality; finite-horizon backward induction; average-reward criteria.

Further reading: Howard, Dynamic Programming and Markov Processes (MIT Press, 1960), which introduced policy iteration; Puterman and Shin, Modified policy iteration algorithms for discounted Markov decision problems (1978); de Farias and Van Roy, The linear programming approach to approximate dynamic programming (2003); Ye, The simplex and policy-iteration methods are strongly polynomial for the Markov decision problem with a fixed discount rate (2011); Scherrer, Improved and generalized upper bounds on the complexity of policy iteration (2016).

3. Multi-armed bandits {#3-multi-armed-bandits}

Topics: The exploration–exploitation dilemma in its simplest form; action-value estimates and incremental updates; regret and its decomposition; greedy, ε-greedy, and optimistic initial values; explore-then-commit; the upper confidence bound and its analysis; concentration inequalities; the Lai–Robbins lower bound and minimax lower bounds; KL-UCB; Thompson sampling; gradient bandits; nonstationary bandits; best-arm identification.

Further reading: Thompson, On the likelihood that one unknown probability exceeds another in view of the evidence of two samples (1933); Lai and Robbins, Asymptotically efficient adaptive allocation rules (1985); Auer, Cesa-Bianchi, and Fischer, Finite-time analysis of the multiarmed bandit problem (2002); Garivier and Cappé, The KL-UCB algorithm for bounded stochastic bandits and beyond (2011); Agrawal and Goyal, Analysis of Thompson sampling for the multi-armed bandit problem (2012); Bubeck and Cesa-Bianchi, Regret analysis of stochastic and nonstochastic multi-armed bandit problems (2012); Russo et al., A tutorial on Thompson sampling (2018); Slivkins, Introduction to multi-armed bandits (2019).

4. Contextual, Bayesian, and adversarial bandits {#4-contextual-bayesian-and-adversarial-bandits}

Topics: Bayesian bandits and the Gittins index; the value of information in bandits; adversarial bandits, exponential weights, and Exp3; contextual bandits and Exp4; linear bandits, least-squares confidence ellipsoids, and LinUCB; linear Thompson sampling; reductions to regression and SquareCB; neural contextual bandits; off-policy evaluation of logged bandit feedback with inverse propensity and doubly robust estimators; bandits in recommendation, advertising, and experimentation.

Further reading: Gittins, Bandit processes and dynamic allocation indices (1979); Weber, On the Gittins index for multiarmed bandits (1992); Auer et al., The nonstochastic multiarmed bandit problem (2002); Li et al., A contextual-bandit approach to personalized news article recommendation (2010); Abbasi-Yadkori, Pál, and Szepesvári, Improved algorithms for linear stochastic bandits (2011); Dudík, Langford, and Li, Doubly robust policy evaluation and learning (2011); Agrawal and Goyal, Thompson sampling for contextual bandits with linear payoffs (2013); Russo and Van Roy, Learning to optimize via posterior sampling (2014); Agarwal et al., Taming the monster (2014); Foster and Rakhlin, Beyond UCB (2020). The Open Bandit Pipeline provides logged bandit data from a fashion e-commerce platform.

5. Monte Carlo methods {#5-monte-carlo-methods}

Topics: Learning values from complete episodes without a model; first-visit and every-visit Monte Carlo and their bias and variance; incremental means and constant step sizes; Monte Carlo estimation of action values; control with exploring starts and with ε-soft policies; off-policy prediction with importance sampling, ordinary and weighted; incremental off-policy Monte Carlo control; discounting-aware and per-decision importance sampling; blackjack as a running example.

Further reading: Singh and Sutton, Reinforcement learning with replacing eligibility traces (1996), which analyzes first-visit and every-visit Monte Carlo; Precup, Sutton, and Singh, Eligibility traces for off-policy policy evaluation (2000); Mahmood, van Hasselt, and Sutton, Weighted importance sampling for off-policy learning with linear function approximation (2014).

6. Temporal-difference learning {#6-temporal-difference-learning}

Topics: Bootstrapping from the next estimate; TD(0) and the TD error; the advantages of TD over Monte Carlo and dynamic programming; random walks and step sizes; batch updating, certainty equivalence, and the optimality of batch TD; stochastic approximation and the convergence of TD; afterstates; TD-Gammon; TD errors, dopamine, and the reward-prediction-error hypothesis.

Further reading: Sutton, Learning to predict by the methods of temporal differences (1988); Dayan, The convergence of TD(λ) for general λ (1992); Jaakkola, Jordan, and Singh, On the convergence of stochastic iterative dynamic programming algorithms (1994); Tesauro, Temporal difference learning and TD-Gammon (1995); Schultz, Dayan, and Montague, A neural substrate of prediction and reward (1997); Bhandari, Russo, and Singal, A finite time analysis of temporal difference learning with linear function approximation (2018).

7. Model-free control {#7-model-free-control}

Topics: Generalized policy iteration without a model; greedy in the limit with infinite exploration; on-policy control with SARSA; off-policy control with Q-learning; expected SARSA as a family that contains both; cliff walking and the cost of exploration; maximization bias and double Q-learning; convergence of SARSA and Q-learning; the windy gridworld, Taxi, and other test problems; action selection with softmax and optimism.

Further reading: Watkins, Learning from delayed rewards (PhD thesis, 1989); Watkins and Dayan, Q-learning (1992); Tsitsiklis, Asynchronous stochastic approximation and Q-learning (1994); Singh et al., Convergence results for single-step on-policy reinforcement-learning algorithms (2000); van Seijen et al., A theoretical and empirical analysis of Expected Sarsa (2009); van Hasselt, Double Q-learning (2010).

8. Multi-step bootstrapping and eligibility traces {#8-multi-step-bootstrapping-and-eligibility-traces}

Topics: n-step returns between TD and Monte Carlo; n-step TD and n-step SARSA; the bias–variance trade-off in the choice of n; the λ-return and the forward view; TD(λ), eligibility traces, and the backward view; accumulating, replacing, and dutch traces; the online λ-return and true online TD(λ); SARSA(λ); variable λ and γ; traces with experience replay.

Further reading: Singh and Sutton, Reinforcement learning with replacing eligibility traces (1996); Peng and Williams, Incremental multi-step Q-learning (1996); van Seijen and Sutton, True online TD(λ) (2014); van Seijen et al., True online temporal-difference learning (2016); De Asis et al., Multi-step reinforcement learning: a unifying algorithm (2018); Daley and Amato, Reconciling λ-returns with experience replay (2019).

9. Off-policy learning {#9-off-policy-learning}

Topics: Behavior and target policies; importance sampling for multi-step returns; the variance of importance weights and the curse of horizon; per-decision importance sampling and control variates; n-step off-policy methods; tree backup and Q(σ) without importance sampling; off-policy traces: Watkins’s Q(λ), tree-backup(λ), and Retrace(λ); emphatic temporal-difference learning; off-policy evaluation with doubly robust and marginalized estimators.

Further reading: Precup, Sutton, and Singh, Eligibility traces for off-policy policy evaluation (2000); Munos et al., Safe and efficient off-policy reinforcement learning (2016); Harutyunyan et al., Q(λ) with off-policy corrections (2016); Sutton, Mahmood, and White, An emphatic approach to the problem of off-policy temporal-difference learning (2016); Jiang and Li, Doubly robust off-policy value evaluation for reinforcement learning (2016); Thomas and Brunskill, Data-efficient off-policy policy evaluation for reinforcement learning (2016); Liu et al., Breaking the curse of horizon (2018); Uehara, Shi, and Kallus, A review of off-policy evaluation in reinforcement learning (2022).

10. Planning and learning with tabular models {#10-planning-and-learning-with-tabular-models}

Topics: Models as distributions or as samples; Dyna and simulated experience; learning with a wrong or changing model, Dyna-Q+; prioritized sweeping; expected versus sample updates; trajectory sampling and real-time dynamic programming; decision-time planning, heuristic search, and rollout algorithms; sparse sampling; Monte Carlo tree search and UCT in MDPs; when parametric models are better than replay.

Further reading: Sutton, Dyna, an integrated architecture for learning, planning, and reacting (1991); Moore and Atkeson, Prioritized sweeping (1993); Barto, Bradtke, and Singh, Learning to act using real-time dynamic programming (1995); Tesauro and Galperin, On-line policy improvement using Monte-Carlo search (1996); Kearns, Mansour, and Ng, A sparse sampling algorithm for near-optimal planning in large Markov decision processes (2002); Kocsis and Szepesvári, Bandit based Monte-Carlo planning (2006); Browne et al., A survey of Monte Carlo tree search methods (2012); van Hasselt, Hessel, and Aslanides, When to use parametric models in reinforcement learning? (2019).

11. Value function approximation {#11-value-function-approximation}

Topics: Generalization across states; the mean squared value error and the on-policy distribution; stochastic-gradient and semi-gradient methods; linear methods and the TD fixed point; feature construction: polynomials, Fourier bases, coarse coding, tile coding, and radial basis functions; step sizes for linear methods; least-squares TD and LSPI; episodic semi-gradient control and the mountain car; the average-reward setting for continuing tasks; neural networks as features, previewed.

Further reading: Sutton, Generalization in reinforcement learning: successful examples using sparse coarse coding (1995); Bradtke and Barto, Linear least-squares algorithms for temporal difference learning (1996); Tsitsiklis and Van Roy, An analysis of temporal-difference learning with function approximation (1997); Boyan, Technical update: least-squares temporal difference learning (2002); Lagoudakis and Parr, Least-squares policy iteration (2003); Geramifard et al., A tutorial on linear function approximators for dynamic programming and reinforcement learning (2013); Dann, Neumann, and Peters, Policy evaluation with temporal differences: a survey and comparison (2014).

12. The deadly triad and gradient-TD methods {#12-the-deadly-triad-and-gradient-td-methods}

Topics: Off-policy learning with function approximation; Baird’s counterexample and divergence; the deadly triad of function approximation, bootstrapping, and off-policy learning; the geometry of linear value functions; the Bellman error, the projected Bellman error, and why the Bellman error is not learnable; residual gradients; gradient-TD methods, GTD2 and TDC; emphatic TD; batch reinforcement learning, fitted value and Q-iteration, and their error bounds; the deadly triad in deep networks.

Further reading: Baird, Residual algorithms: reinforcement learning with function approximation (1995); Ernst, Geurts, and Wehenkel, Tree-based batch mode reinforcement learning (2005); Riedmiller, Neural fitted Q iteration (2005); Munos and Szepesvári, Finite-time bounds for fitted value iteration (2008); Sutton, Maei, and Szepesvári, A convergent O(n) temporal-difference algorithm for off-policy learning with linear function approximation (2008); Sutton et al., Fast gradient-descent methods for temporal-difference learning with linear function approximation (2009); van Hasselt et al., Deep reinforcement learning and the deadly triad (2018); Ghiassian et al., Gradient temporal-difference learning with regularized corrections (2020); Fujimoto et al., Why should I trust you, Bellman? (2022).

13. Policy gradient and actor-critic methods {#13-policy-gradient-and-actor-critic-methods}

Topics: Parameterized policies and why to learn them directly; the policy gradient theorem, episodic and continuing; the score-function estimator and REINFORCE; baselines and variance reduction; actor-critic methods with one-step and λ-returns; compatible function approximation; the natural policy gradient and the Fisher information; natural actor-critic; Gaussian policies for continuous actions; deterministic policy gradients; off-policy actor-critic.

Further reading: Barto, Sutton, and Anderson, Neuronlike adaptive elements that can solve difficult learning control problems (1983); Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning (1992); Sutton et al., Policy gradient methods for reinforcement learning with function approximation (1999); Konda and Tsitsiklis, Actor-critic algorithms (1999); Kakade, A natural policy gradient (2001); Greensmith, Bartlett, and Baxter, Variance reduction techniques for gradient estimates in reinforcement learning (2004); Peters and Schaal, Natural actor-critic (2008); Degris, White, and Sutton, Off-policy actor-critic (2012); Silver et al., Deterministic policy gradient algorithms (2014). OpenAI Spinning Up’s introduction to policy optimization derives the estimators step by step.

14. Partially observable environments {#14-partially-observable-environments}

Topics: Observations, histories, and the failure of the Markov property; partially observable MDPs; belief states and Bayesian filtering; the belief MDP; piecewise-linear convex value functions and α-vectors; exact value iteration and its cost; point-based value iteration; online planning with POMCP and DESPOT; QMDP and other approximations; memoryless and finite-state controllers; agent state, recurrent policies, and predictive state representations.

Further reading: Smallwood and Sondik, The optimal control of partially observable Markov processes over a finite horizon (1973); Kaelbling, Littman, and Cassandra, Planning and acting in partially observable stochastic domains (1998); Littman, Sutton, and Singh, Predictive representations of state (2001); Pineau, Gordon, and Thrun, Point-based value iteration: an anytime algorithm for POMDPs (2003); Spaan and Vlassis, Perseus: randomized point-based value iteration for POMDPs (2005); Silver and Veness, Monte-Carlo planning in large POMDPs (2010); Shani, Pineau, and Kaplow, A survey of point-based POMDP solvers (2013); Ye et al., DESPOT: online POMDP planning with regularization (2017); Ni, Eysenbach, and Salakhutdinov, Recurrent model-free RL can be a strong baseline for many POMDPs (2022).

15. Optimal control and trajectory optimization {#15-optimal-control-and-trajectory-optimization}

Topics: Continuous states and actions with known dynamics; the linear–quadratic regulator by dynamic programming and the Riccati equation; infinite-horizon LQR and stability; LQG and certainty equivalence; linearization, iterative LQR, and differential dynamic programming; shooting and collocation methods; sampling-based optimizers: random shooting, the cross-entropy method, and MPPI; model-predictive control and receding horizons; how optimal control relates to reinforcement learning.

Further reading: Mayne, A second-order gradient method for determining optimal trajectories of non-linear discrete-time systems (1966), the origin of differential dynamic programming; Betts, Survey of numerical methods for trajectory optimization (1998); Mayne et al., Constrained model predictive control: stability and optimality (2000); de Boer et al., A tutorial on the cross-entropy method (2005); Tassa, Erez, and Todorov, Synthesis and stabilization of complex behaviors through online trajectory optimization (2012); Kelly, An introduction to trajectory optimization: how to do your own direct collocation (2017); Williams et al., Information theoretic MPC for model-based reinforcement learning (2017); Fazel et al., Global convergence of policy gradient methods for the linear quadratic regulator (2018); Recht, A tour of reinforcement learning: the view from continuous control (2019); Howell et al., Predictive sampling: real-time behaviour synthesis with MuJoCo (2022).

Part II — Modern reinforcement learning. Blocks 16–31 follow CS285, CS224R, and the research literature to the current frontier.

16. Deep Q-learning {#16-deep-q-learning}

Topics: Q-learning with neural networks and why it is unstable; experience replay and target networks; DQN on Atari and the Arcade Learning Environment; evaluation protocols, sticky actions, and statistical reporting; overestimation and double DQN; dueling architectures; prioritized replay; multi-step returns; noisy networks; Rainbow and ablations; replay ratio and the fundamentals of replay.

Further reading: Lin, Self-improving reactive agents based on reinforcement learning, planning and teaching (1992); Bellemare et al., The Arcade Learning Environment (2013); Mnih et al., Human-level control through deep reinforcement learning (2015); van Hasselt, Guez, and Silver, Deep reinforcement learning with double Q-learning (2016); Wang et al., Dueling network architectures for deep reinforcement learning (2016); Schaul et al., Prioritized experience replay (2016); Fortunato et al., Noisy networks for exploration (2018); Hessel et al., Rainbow (2018); Machado et al., Revisiting the Arcade Learning Environment (2018); Fedus et al., Revisiting fundamentals of experience replay (2020); Agarwal et al., Deep reinforcement learning at the edge of the statistical precipice (2021). CleanRL has single-file implementations of DQN and Rainbow.

17. Distributional reinforcement learning {#17-distributional-reinforcement-learning}

Topics: The random return and its distribution; the distributional Bellman equation; Wasserstein and Cramér metrics and contraction; categorical representations and C51; quantile representations, quantile regression, QR-DQN, IQN, and FQF; statistical functionals and what can be learned exactly; distributional control and its instability; why distributional agents learn better representations; risk-sensitive control; distributional codes for value in dopamine neurons; value learning by classification.

Further reading: Bellemare, Dabney, and Munos, A distributional perspective on reinforcement learning (2017); Dabney et al., Distributional reinforcement learning with quantile regression (2018); Dabney et al., Implicit quantile networks for distributional reinforcement learning (2018); Rowland et al., An analysis of categorical distributional reinforcement learning (2018); Rowland et al., Statistics and samples in distributional reinforcement learning (2019); Lyle, Castro, and Bellemare, A comparative analysis of expected and distributional reinforcement learning (2019); Yang et al., Fully parameterized quantile function for distributional reinforcement learning (2019); Dabney et al., A distributional code for value in dopamine-based reinforcement learning (2020); Farebrother et al., Stop regressing: training value functions via classification for scalable deep RL (2024). CleanRL implements C51.

18. Data-efficient and scalable value-based agents {#18-data-efficient-and-scalable-value-based-agents}

Topics: Sample efficiency and the Atari 100k benchmark; data augmentation and self-predictive representations; the replay ratio; loss of plasticity, the primacy bias, dormant neurons, and resets; network architecture and normalization for value learning; scaling parameters with regularization, mixtures of experts, and pruning; BBF and human-level efficiency; parallel Q-learning without replay; predictable scaling of value-based RL and compute-optimal allocation; very deep networks for self-supervised RL.

Further reading: Kaiser et al., Model-based reinforcement learning for Atari (2020), which introduced Atari 100k; Kostrikov, Yarats, and Fergus, Image augmentation is all you need (2021); Schwarzer et al., Data-efficient reinforcement learning with self-predictive representations (2021); Nikishin et al., The primacy bias in deep reinforcement learning (2022); Lyle, Rowland, and Dabney, Understanding and preventing capacity loss in reinforcement learning (2022); Sokar et al., The dormant neuron phenomenon in deep reinforcement learning (2023); Schwarzer et al., Bigger, better, faster: human-level Atari with human-level efficiency (2023); Dohare et al., Loss of plasticity in deep continual learning (2024); Obando-Ceron et al., Mixtures of experts unlock parameter scaling for deep RL (2024); Nauman et al., Bigger, regularized, optimistic (2024); Lee et al., SimBa: simplicity bias for scaling up parameters in deep reinforcement learning (2025); Gallici et al., Simplifying deep temporal difference learning (2025); Rybkin et al., Value-based deep RL scales predictably (2025); Fu et al., Compute-optimal scaling for value-based deep RL (2025); Wang et al., 1000 layer networks for self-supervised RL (2025). CleanRL implements PQN.

19. Deep actor-critic and distributed RL {#19-deep-actor-critic-and-distributed-rl}

Topics: Advantage actor-critic with neural networks; parallel environments and synchronous versus asynchronous updates; generalized advantage estimation; entropy regularization; shared and separate networks; IMPALA, V-trace, and off-policy correction of stale data; distributed replay with Ape-X and recurrent replay with R2D2; Never Give Up and Agent57; accelerator-centric architectures, from SEED to Podracer and GPU simulation; large-scale systems such as OpenAI Five.

Further reading: Nair et al., Massively parallel methods for deep reinforcement learning (2015); Mnih et al., Asynchronous methods for deep reinforcement learning (2016); Schulman et al., High-dimensional continuous control using generalized advantage estimation (2016); Espeholt et al., IMPALA (2018); Horgan et al., Distributed prioritized experience replay (2018); Kapturowski et al., Recurrent experience replay in distributed reinforcement learning (ICLR, 2019); Berner et al., Dota 2 with large scale deep reinforcement learning (2019); Badia et al., Agent57 (2020); Espeholt et al., SEED RL (2020); Hessel et al., Podracer architectures for scalable reinforcement learning (2021); Rudin et al., Learning to walk in minutes using massively parallel deep reinforcement learning (2021); Huang et al., A2C is a special case of PPO (2022). Gymnasium’s tutorial on vectorized A2C and PureJaxRL show parallel training in practice.

20. Trust regions and proximal policy optimization {#20-trust-regions-and-proximal-policy-optimization}

Topics: Why large policy steps fail; the performance difference lemma and monotonic improvement; conservative policy iteration; trust regions, TRPO, and conjugate gradients; PPO’s clipped objective and KL penalties; the implementation details that matter and what the ablations show; value clipping, normalization, and minibatches; phasic policy gradient; maximum a posteriori policy optimization; mirror descent and regularized MDPs; PPO as the default on-policy algorithm.

Further reading: Kakade and Langford, Approximately optimal approximate reinforcement learning (2002); Schulman et al., Trust region policy optimization (2015); Schulman et al., Proximal policy optimization algorithms (2017); Abdolmaleki et al., Maximum a posteriori policy optimisation (2018); Geist, Scherrer, and Pietquin, A theory of regularized Markov decision processes (2019); Engstrom et al., Implementation matters in deep policy gradients (2020); Andrychowicz et al., What matters in on-policy reinforcement learning? (2021); Cobbe et al., Phasic policy gradient (2021); Huang et al., The 37 implementation details of proximal policy optimization (2022); Moalla et al., No representation, no trust (2024). CleanRL documents its PPO variants line by line.

21. Continuous control and maximum-entropy RL {#21-continuous-control-and-maximum-entropy-rl}

Topics: Continuous actions and Gaussian and squashed policies; deterministic policy gradients and DDPG; overestimation in actor-critic and TD3; the maximum-entropy objective, soft value functions, and soft Q-learning; soft actor-critic and automatic temperature tuning; RL as probabilistic inference; linearly solvable MDPs; ensembles, high update-to-data ratios, and normalization in critics: REDQ, DroQ, CrossQ, TD7, and their successors; benchmark suites, seeds, and reproducibility.

Further reading: Todorov, Linearly-solvable Markov decision problems (2006); Lillicrap et al., Continuous control with deep reinforcement learning (2016); Haarnoja et al., Reinforcement learning with deep energy-based policies (2017); Fujimoto, van Hoof, and Meger, Addressing function approximation error in actor-critic methods (2018); Haarnoja et al., Soft actor-critic (2018) and Soft actor-critic algorithms and applications (2018); Levine, Reinforcement learning and control as probabilistic inference (2018); Tassa et al., DeepMind Control Suite (2018); Henderson et al., Deep reinforcement learning that matters (2018); Chen et al., Randomized ensembled double Q-learning (2021); Bhatt et al., CrossQ (2024); Fujimoto et al., Towards general-purpose model-free reinforcement learning (2025); Seo et al., FastTD3 (2025).

22. Exploration in deep RL {#22-exploration-in-deep-rl}

Topics: Why dithering fails in deep and sparse problems; deep exploration and the chain problem; optimism in MDPs; count-based bonuses and pseudo-counts; intrinsic motivation, curiosity, and prediction error; random network distillation; the noisy-TV problem; episodic novelty and Never Give Up; posterior sampling, bootstrapped DQN, randomized priors, and epistemic networks; information gain; hard-exploration games and Go-Explore; exploration without rewards.

Further reading: Oudeyer and Kaplan, What is intrinsic motivation? A typology of computational approaches (2007); Schmidhuber, Formal theory of creativity, fun, and intrinsic motivation (2010); Osband et al., Deep exploration via bootstrapped DQN (2016); Bellemare et al., Unifying count-based exploration and intrinsic motivation (2016); Pathak et al., Curiosity-driven exploration by self-supervised prediction (2017); Osband, Aslanides, and Cassirer, Randomized prior functions for deep reinforcement learning (2018); Burda et al., Exploration by random network distillation (2019); Osband et al., Deep exploration via randomized value functions (2019); Badia et al., Never give up (2020); Ecoffet et al., First return, then explore (2021); Henaff et al., Exploration via elliptical episodic bonuses (2022); Lu et al., Reinforcement learning, bit by bit (2023).

23. Model-based RL and world models {#23-model-based-rl-and-world-models}

Topics: Learning dynamics models and why errors compound; uncertainty-aware ensembles and PETS; model-based policy optimization and branched rollouts; latent world models: world models, PlaNet, and the Dreamer agents; learning behaviors in imagination; value-equivalent and decision-aware models; TD-MPC and planning with learned latent models; transformer and diffusion world models; generative interactive environments; training agents inside world models; when models help and when they hurt.

Further reading: Ha and Schmidhuber, World models (2018); Chua et al., Deep reinforcement learning in a handful of trials using probabilistic dynamics models (2018); Hafner et al., Learning latent dynamics for planning from pixels (2019); Janner et al., When to trust your model (2019); Hafner et al., Dream to control (2020); Hansen, Wang, and Su, Temporal difference learning for model predictive control (2022); Micheli, Alonso, and Fleuret, Transformers are sample-efficient world models (2023); Hansen, Su, and Wang, TD-MPC2 (2024); Alonso et al., Diffusion for world modeling: visual details matter in Atari (2024); Bruce et al., Genie: generative interactive environments (2024); Hafner et al., Mastering diverse control tasks through world models (2025), DreamerV3; Hafner, Yan, and Lillicrap, Training agents inside of scalable world models (2025); DeepMind, Genie 3: a new frontier for world models (2025).

24. Planning with learned models, AlphaZero and MuZero {#24-planning-with-learned-models-alphazero-and-muzero}

Topics: Search as policy improvement; expert iteration; AlphaGo, AlphaGo Zero, and AlphaZero: PUCT, self-play, and training targets; MuZero: planning with a value-equivalent learned model; MCTS as regularized policy optimization; Gumbel search with few simulations; sampled and stochastic MuZero; data-efficient search agents, EfficientZero; offline search with MuZero Unplugged; search beyond games: matrix multiplication, sorting, and formal mathematics.

Further reading: Silver et al., Mastering the game of Go with deep neural networks and tree search (2016); Anthony, Tian, and Barber, Thinking fast and slow with deep learning and tree search (2017); Silver et al., Mastering the game of Go without human knowledge (2017); Silver et al., A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play (2018); Schrittwieser et al., Mastering Atari, Go, chess and shogi by planning with a learned model (2020); Grill et al., Monte-Carlo tree search as regularized policy optimization (2020); Hubert et al., Learning and planning in complex action spaces (2021); Ye et al., Mastering Atari games with limited data (2021); Danihelka et al., Policy improvement by planning with Gumbel (2022); Fawzi et al., Discovering faster matrix multiplication algorithms with reinforcement learning (2022); Mankowitz et al., Faster sorting algorithms discovered using deep reinforcement learning (2023); Wang et al., EfficientZero V2 (2024); Hubert et al., Olympiad-level formal mathematical reasoning with reinforcement learning (2025). DeepMind’s mctx implements AlphaZero, MuZero, and Gumbel MuZero search.

25. Imitation learning and inverse RL {#25-imitation-learning-and-inverse-rl}

Topics: Behavior cloning and compounding errors; DAgger and interactive imitation; imitation as distribution matching; inverse reinforcement learning, its ambiguity, and feature matching; maximum-entropy IRL; adversarial imitation, GAIL, and AIRL; imitation with soft Q-functions, SQIL and IQ-Learn; moment matching and IRL without inner RL loops; expressive policies for multimodal demonstrations: action chunking and diffusion policies; imitation at scale from video and internet data; reward learning from human feedback, previewed.

Further reading: Pomerleau, ALVINN: an autonomous land vehicle in a neural network (1988); Ng and Russell, Algorithms for inverse reinforcement learning (2000); Abbeel and Ng, Apprenticeship learning via inverse reinforcement learning (2004); Ziebart et al., Maximum entropy inverse reinforcement learning (2008); Ross, Gordon, and Bagnell, A reduction of imitation learning and structured prediction to no-regret online learning (2011); Ho and Ermon, Generative adversarial imitation learning (2016); Fu, Luo, and Levine, Learning robust rewards with adversarial inverse reinforcement learning (2018); Osa et al., An algorithmic perspective on imitation learning (2018); Reddy, Dragan, and Levine, SQIL (2020); Garg et al., IQ-Learn (2021); Swamy et al., Of moments and matching (2021); Baker et al., Video pretraining (2022); Chi et al., Diffusion policy (2023); Zhao et al., Learning fine-grained bimanual manipulation with low-cost hardware (2023).

26. Offline reinforcement learning {#26-offline-reinforcement-learning}

Topics: Learning from fixed datasets; distribution shift and overestimation of out-of-distribution actions; policy constraints: BCQ, BEAR, and TD3+BC; conservative Q-learning; in-sample learning with implicit Q-learning; advantage-weighted regression and AWAC; model-based offline RL and uncertainty penalties; pessimism and coverage in theory; offline RL as sequence modeling and conditional behavior cloning; expressive diffusion and flow policies; off-policy evaluation and model selection; offline-to-online fine-tuning; when to prefer offline RL over behavior cloning; benchmarks.

Further reading: Fujimoto, Meger, and Precup, Off-policy deep reinforcement learning without exploration (2019); Levine et al., Offline reinforcement learning: tutorial, review, and perspectives on open problems (2020); Fu et al., D4RL (2020); Kumar et al., Conservative Q-learning for offline reinforcement learning (2020); Yu et al., MOPO (2020); Jin, Yang, and Wang, Is pessimism provably efficient for offline RL? (2021); Fujimoto and Gu, A minimalist approach to offline reinforcement learning (2021); Chen et al., Decision Transformer (2021); Kostrikov, Nair, and Levine, Offline reinforcement learning with implicit Q-learning (2022); Emmons et al., RvS: what is essential for offline RL via supervised learning? (2022); Nakamoto et al., Cal-QL (2023); Ball et al., Efficient online reinforcement learning with offline data (2023); Park et al., OGBench (2025); Park, Li, and Levine, Flow Q-learning (2025). Minari hosts the standard datasets and CORL gives single-file implementations.

27. Multi-agent RL and self-play {#27-multi-agent-rl-and-self-play}

Topics: Stochastic games and their solution concepts; independent learners and non-stationarity; minimax-Q and Nash-Q; centralized training with decentralized execution; value decomposition with VDN and QMIX; multi-agent policy gradients, COMA, and MAPPO; self-play, fictitious play, and population-based training; policy-space response oracles; counterfactual regret minimization, DeepStack, Libratus, Pluribus, and ReBeL; AlphaStar and league training; DeepNash and Cicero; zero-shot coordination and Hanabi; emergent communication, social dilemmas, and autocurricula.

Further reading: Littman, Markov games as a framework for multi-agent reinforcement learning (1994); Hu and Wellman, Nash Q-learning for general-sum stochastic games (2003); Zinkevich et al., Regret minimization in games with incomplete information (2007); Heinrich and Silver, Deep reinforcement learning from self-play in imperfect-information games (2016); Lanctot et al., A unified game-theoretic approach to multiagent reinforcement learning (2017); Lowe et al., Multi-agent actor-critic for mixed cooperative-competitive environments (2017); Moravčík et al., DeepStack (2017); Rashid et al., QMIX (2018); Brown and Sandholm, Superhuman AI for multiplayer poker (2019); Vinyals et al., Grandmaster level in StarCraft II using multi-agent reinforcement learning (2019); Brown et al., Combining deep reinforcement learning and search for imperfect-information games (2020); Hu et al., Other-play for zero-shot coordination (2020); Baker et al., Emergent tool use from multi-agent autocurricula (2020); Yu et al., The surprising effectiveness of PPO in cooperative multi-agent games (2022); Perolat et al., Mastering the game of Stratego with model-free multiagent reinforcement learning (2022); Meta FAIR Diplomacy Team, Human-level play in the game of Diplomacy by combining language models with strategic reasoning (2022). PettingZoo, OpenSpiel, and JaxMARL provide environments and algorithms.

28. Reinforcement learning for language models and reasoning {#28-reinforcement-learning-for-language-models-and-reasoning}

Topics: The language model as a policy: token-level and sequence-level MDPs; KL-regularized RL and its optimal policy; PPO for language models with a learned critic; critic-free estimators: REINFORCE with leave-one-out baselines, GRPO, and their biases; estimating and differentiating the KL term; importance ratios at the token and sequence level and GSPO; clipping, length normalization, and the fixes of Dr. GRPO and DAPO; entropy collapse and exploration; does RL expand what the base model can solve; outcome and process rewards; asynchronous and off-policy training systems; multi-turn and agentic RL; scaling RL compute.

Further reading: Schulman, Approximating KL divergence (2020); Lightman et al., Let’s verify step by step (2023); Shao et al., DeepSeekMath (2024), which introduced GRPO; Ahmadian et al., Back to basics: revisiting REINFORCE style optimization for learning from human feedback in LLMs (2024); Rafailov et al., From r to Q*: your language model is secretly a Q-function (2024); Zhou et al., ArCHer (2024); DeepSeek-AI, DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning (2025); Kimi Team, Kimi k1.5 (2025); Liu et al., Understanding R1-Zero-like training: a critical perspective (2025); Yu et al., DAPO (2025); Kazemnejad et al., VinePPO (2025); Yue et al., Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? (2025); Cui et al., The entropy mechanism of reinforcement learning for reasoning language models (2025); Tang and Munos, On a few pitfalls in KL divergence gradient estimation for RL (2025); Zheng et al., Group sequence policy optimization (2025); Noukhovitch et al., Asynchronous RLHF (2025); Fu et al., AReaL (2025); Wei et al., SWE-RL (2025); Khatri et al., The art of scaling reinforcement learning compute for LLMs (2025); Zhang et al., A survey of reinforcement learning for large reasoning models (2025). TRL’s GRPOTrainer and verl implement these algorithms.

29. Generalist agents, meta-RL, and open-endedness {#29-generalist-agents-meta-rl-and-open-endedness}

Topics: Generalization across tasks, levels, and goals; goal-conditioned policies and universal value functions; unsupervised skill discovery; meta-reinforcement learning: learning to learn with recurrent policies, gradient-based adaptation, and task inference; Bayes-adaptive MDPs; in-context reinforcement learning and algorithm distillation; generalist sequence-model agents; RL fine-tuning of vision–language–action models; automatic curricula and unsupervised environment design; open-endedness, interestingness, and foundation models as environment generators.

Further reading: Schaul et al., Universal value function approximators (2015); Duan et al., RL²: fast reinforcement learning via slow reinforcement learning (2016); Wang et al., Learning to reinforcement learn (2016); Finn, Abbeel, and Levine, Model-agnostic meta-learning for fast adaptation of deep networks (2017); Eysenbach et al., Diversity is all you need (2019); Wang et al., Paired open-ended trailblazer (2019); Rakelly et al., Efficient off-policy meta-reinforcement learning via probabilistic context variables (2019); Zintgraf et al., VariBAD (2020); Dennis et al., Emergent complexity and zero-shot transfer via unsupervised environment design (2020); Jiang, Grefenstette, and Rocktäschel, Prioritized level replay (2021); Reed et al., A generalist agent (2022); Eysenbach et al., Contrastive learning as goal-conditioned reinforcement learning (2022); Laskin et al., In-context reinforcement learning with algorithm distillation (2023); Adaptive Agent Team, Human-timescale adaptation in an open-ended task space (2023); Zhang et al., OMNI: open-endedness via models of human notions of interestingness (2024); Hughes et al., Open-endedness is essential for artificial superhuman intelligence (2024); Li et al., SimpleVLA-RL (2025).

30. The theory of reinforcement learning {#30-the-theory-of-reinforcement-learning}

Topics: Sample complexity with a generative model; PAC and regret frameworks for exploration; optimism in tabular MDPs, UCRL2 and UCBVI; posterior sampling; lower bounds; the efficiency of Q-learning; linear function approximation, linear MDPs, and when realizability is not enough; Bellman rank, eluder dimensions, and the decision-estimation coefficient; offline RL, coverage, and pessimism; convergence of policy gradient, natural policy gradient, and policy mirror descent; the statistical cost of long horizons.

Further reading: Kearns and Singh, Near-optimal reinforcement learning in polynomial time (2002); Brafman and Tennenholtz, R-MAX (2002); Strehl, Li, and Littman, Reinforcement learning in finite MDPs: PAC analysis (2009); Jaksch, Ortner, and Auer, Near-optimal regret bounds for reinforcement learning (2010); Azar, Munos, and Kappen, Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model (2013); Osband, Russo, and Van Roy, (More) efficient reinforcement learning via posterior sampling (2013); Azar, Osband, and Munos, Minimax regret bounds for reinforcement learning (2017); Jiang et al., Contextual decision processes with low Bellman rank are PAC-learnable (2017); Jin et al., Is Q-learning provably efficient? (2018); Du et al., Is a good representation sufficient for sample efficient reinforcement learning? (2020); Jin et al., Provably efficient reinforcement learning with linear function approximation (2020); Mei et al., On the global convergence rates of softmax policy gradient methods (2020); Weisz, Amortila, and Szepesvári, Exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions (2021); Agarwal et al., On the theory of policy gradient methods (2021); Jin, Liu, and Miryoosefi, Bellman eluder dimension (2021); Foster et al., The statistical complexity of interactive decision making (2021); Cen et al., Fast global convergence of natural policy gradient methods with entropy regularization (2022); Lan, Policy mirror descent for reinforcement learning (2023).

31. Reinforcement learning in the real world {#31-reinforcement-learning-in-the-real-world}

Topics: Reward design, shaping, and specification; potential-based shaping; sparse rewards, goal-conditioned RL, and hindsight relabeling; temporal abstraction with options, feudal and hierarchical RL; constrained MDPs and safe exploration; simulation, system identification, and domain randomization; sim-to-real transfer in locomotion and manipulation; rapid adaptation; deployed systems: data-center cooling, chip floorplanning, stratospheric balloons, plasma control, racing cars, and drones; the challenges of real-world RL.

Further reading: Dayan and Hinton, Feudal reinforcement learning (1992); Ng, Harada, and Russell, Policy invariance under reward transformations (1999); Sutton, Precup, and Singh, Between MDPs and semi-MDPs (1999); García and Fernández, A comprehensive survey on safe reinforcement learning (2015); Amodei et al., Concrete problems in AI safety (2016); Bacon, Harb, and Precup, The option-critic architecture (2017); Achiam et al., Constrained policy optimization (2017); Andrychowicz et al., Hindsight experience replay (2017); Tobin et al., Domain randomization for transferring deep neural networks from simulation to the real world (2017); Peng et al., Sim-to-real transfer of robotic control with dynamics randomization (2018); Lazic et al., Data center cooling using model-predictive control (2018); OpenAI et al., Solving Rubik’s cube with a robot hand (2019); Bellemare et al., Autonomous navigation of stratospheric balloons using reinforcement learning (2020); Kumar et al., RMA: rapid motor adaptation for legged robots (2021); Mirhoseini et al., A graph placement methodology for fast chip design (2021); Dulac-Arnold et al., Challenges of real-world reinforcement learning (2021); Degrave et al., Magnetic control of tokamak plasmas through deep reinforcement learning (2022); Wurman et al., Outracing champion Gran Turismo drivers with deep reinforcement learning (2022); Skalse et al., Defining and characterizing reward hacking (2022); Kaufmann et al., Champion-level drone racing using deep reinforcement learning (2023); Haarnoja et al., Learning agile soccer skills for a bipedal robot with deep reinforcement learning (2024).

Labs and exercises

Every chapter ends with exercises, mixing proofs with experiments on the chapter’s code, and each has a worked solution in a collapsed callout. The Labs folder holds larger projects, each a page with goals, instructions, expected results, and extensions, and a runnable script in Labs/code that reproduces the expected results on a laptop CPU. The labs use Gymnasium environments and MinAtar (Young and Tian, 2019), miniature versions of five Atari games on a \(10\times10\) grid that make deep RL experiments feasible without a GPU.

Connections to later modules

Safety and Frontier takes up reward misspecification and reward hacking, goal misgeneralization, the risks of agents that pursue goals in the world, and the oversight of systems trained with reinforcement learning, including language models trained with human feedback and verifiable rewards. The algorithms, their analysis, and their applications to control, games, decision-making, and language models remain in this module.