Figure sources

Every figure in the RL chapters is an original plot generated for these notes between 24 and 26 September 2026 with NumPy, SciPy, Matplotlib, PyTorch, and Gymnasium; no external artwork or course slides are reproduced. Each chapter’s figures come from one script in Sources/Figure code, which is the complete record of the construction. The shared module rlfig.py sets the palette, fonts, and export (white background, 200 dpi PNG). All environments and data are defined in the scripts: small tabular MDPs (random MDPs, chains, the short corridor, Baird’s counterexample, the blind cliffwalk, Deep Sea, and RiverSwim), gridworlds (a world with two goals, cliff walking, the Dyna maze with its blocking and shortcut variants, a constrained gridworld, and random terrain maps), bandits (the 10-armed testbed and Bernoulli, linear contextual, logged-data, and dependent two-armed bandits), random walks of 5, 19, and 1000 states, the gambler’s problem, blackjack with an infinite deck, the tiger and heaven-and-hell POMDPs, mountain car, cart-poles and pendulums integrated in NumPy, tic-tac-toe, a biased rock–paper–scissors, toy policies over a few language-model responses, and small continuous tasks (a car on a circular track, a two-dimensional state space with a noisy TV, a deceptive maze, and one-step problems with continuous actions). The one external environment is Gymnasium’s CartPole-v1, used only by ch16_dqn.py; no script imports MuJoCo. SciPy is used by ch14_pomdp.py (linprog), ch15_control.py (solve_discrete_are), and ch17_distributional.py (scipy.stats), and PyTorch by the scripts of chapters 16, 19, 22, 23, 26, and 29. No script loads a file or depends on another apart from rlfig.py, and every model is trained from scratch when its script runs; several scripts repeat the setup of code in their chapter, as their docstrings say, but none imports it. The tables below summarize the environments, seeds, and settings of each figure so that it can be read and regenerated without opening the script; the computing setup gives the environment and commands.

Seeds refer to numpy.random.default_rng (“NumPy seed”), torch.manual_seed (“torch seed”), or torch.Generator().manual_seed (“generator seed”); Gymnasium’s reset(seed=...) in chapter 16 and the generators seeded from Python’s hash in chapter 24 are named where they are used. Where a script creates a generator for each run, the runs are identified by their seeds (“NumPy seeds 0–49” means one generator per run); otherwise draws without their own generator, such as exploration, minibatches, and sampled transitions, continue the stream of the script’s generator in the order of the script, so a figure’s random numbers depend on everything drawn before it in the same script. Figures and panels computed without random draws are marked “no randomness”. No script reads command-line arguments; each always writes all its figures. The scripts that use PyTorch set torch.set_num_threads(1), ch16_dqn.py before its second figure, the first to use PyTorch, and ch19_actor_critic.py also sets OPENBLAS_NUM_THREADS=1, unless it is already set, before importing NumPy. According to their docstrings, ch01_mdp.py, ch12_triad.py, ch17_distributional.py, ch20_ppo.py, ch21_maxent.py, and ch27_games.py take a few seconds, ch23_models.py, ch25_imitation.py, and ch28_llm.py about ten seconds, ch14_pomdp.py about fifteen, ch15_control.py about twenty, ch18_rainbow.py and ch26_offline.py about half a minute, ch16_dqn.py about a minute and a half, ch10_planning.py and ch22_exploration.py about two minutes, ch11_fa.py about three, and the others (chapters 2–9, 13, 19, 24, and 29–31) about a minute each. Learning curves average the stated number of runs, and routes, paths, and trajectories come from the single run stated; they illustrate the claims in the notes and are not benchmarks.

1. Markov decision processes {#1-markov-decision-processes}

Script: ch01_mdp.py.

Local figure	Data and construction
rl-mdp-discount.png	No randomness. A \(6\times9\) gridworld (rows counted from the top, from 0) with a wall in column 4, rows 1–4, and two absorbing goals, \(+1\) at row 4, column 1 and \(+10\) in the top-right corner, each paid on entering and followed by zero reward. Four actions (up, down, right, left); the intended move happens with probability 0.9 and each of the other three with probability \(0.1/3\), and a move off the grid or into the wall leaves the agent in place. Value iteration until the largest change is below \(10^{-12}\), for \(\gamma=0.5\), 0.8, and 0.95. Each of the 48 cells that are not goals shows its greedy action and optimal value and is shaded blue if following the greedy actions without slips reaches \(+10\), orange otherwise; the titles count the cells that head for \(+10\). The script prints these counts and the optimal value of the bottom-left corner.
rl-mdp-tiger.png	No randomness. The tiger POMDP as a repeated task: listening costs 1 and hears the tiger’s side correctly with probability 0.85; opening the tiger’s door costs 100 and the other door pays 10; opening places the tiger behind either door with probability 1/2 and gives an uninformative observation; \(\gamma=0.95\). Exact alpha-vector backups for horizons 1 to 30 from the zero vector, keeping after each backup the vectors that are maximal at one of 2,001 evenly spaced beliefs. Left: \(V_t(b)\) for \(t=1\), 2, 4, and 30, colored by the optimal first action (listen green, open left red, open right blue), more opaque for longer horizons. Right: the optimal first action at every grid belief for each horizon from 1 to 30, with the boundaries between actions in black. The script prints the number of alpha-vectors at horizon 30, the listening intervals at horizons 1 and 30, and \(V_{30}(0.5)\).

rl-mdp-discount.png

No randomness. A \(6\times9\) gridworld (rows counted from the top, from 0) with a wall in column 4, rows 1–4, and two absorbing goals, \(+1\) at row 4, column 1 and \(+10\) in the top-right corner, each paid on entering and followed by zero reward. Four actions (up, down, right, left); the intended move happens with probability 0.9 and each of the other three with probability \(0.1/3\), and a move off the grid or into the wall leaves the agent in place. Value iteration until the largest change is below \(10^{-12}\), for \(\gamma=0.5\), 0.8, and 0.95. Each of the 48 cells that are not goals shows its greedy action and optimal value and is shaded blue if following the greedy actions without slips reaches \(+10\), orange otherwise; the titles count the cells that head for \(+10\). The script prints these counts and the optimal value of the bottom-left corner.

rl-mdp-tiger.png

No randomness. The tiger POMDP as a repeated task: listening costs 1 and hears the tiger’s side correctly with probability 0.85; opening the tiger’s door costs 100 and the other door pays 10; opening places the tiger behind either door with probability 1/2 and gives an uninformative observation; \(\gamma=0.95\). Exact alpha-vector backups for horizons 1 to 30 from the zero vector, keeping after each backup the vectors that are maximal at one of 2,001 evenly spaced beliefs. Left: \(V_t(b)\) for \(t=1\), 2, 4, and 30, colored by the optimal first action (listen green, open left red, open right blue), more opaque for longer horizons. Right: the optimal first action at every grid belief for each horizon from 1 to 30, with the boundaries between actions in black. The script prints the number of alpha-vectors at horizon 30, the listening intervals at horizons 1 and 30, and \(V_{30}(0.5)\).

2. Dynamic programming {#2-dynamic-programming}

Script: ch02_dp.py.

Local figure	Data and construction
rl-dp-gambler.png	No randomness. The gambler’s problem with probability of heads 0.4 and goal 100: value 1 at capital 100 and 0 at capital 0, stakes from 1 to \(\min(s,100-s)\). In-place value iteration, sweeping capital from 1 to 99, until the largest change in a sweep is below \(10^{-12}\). Left: the value estimates after sweeps 1, 2, and 3 and at convergence. Right: every stake whose expected value is within \(10^{-9}\) of the best (red dots), over bold play, \(\min(s,100-s)\), as a thick light gray line. The script prints the number of sweeps and of optimal capital–stake pairs.
rl-dp-convergence.png	A random MDP with 200 states and 4 actions (NumPy seed 0): each state–action pair moves to 5 distinct states chosen uniformly, with probabilities from a flat Dirichlet distribution, and has a reward uniform on \([0,1]\). \(v_\) comes from policy iteration with exact evaluation by a linear solve, starting from the first action in every state and stopping when the greedy policy repeats. Value iteration, Gauss–Seidel value iteration (in place, states in order), and modified policy iteration (a greedy step followed by 10 evaluation sweeps) start from \(v=0\) and run until the relative error \(\lVert v-v_\rVert_\infty/\lVert v_*\rVert_\infty\) is below \(10^{-6}\). Left: the relative error at each iteration for \(\gamma=0.99\), with the errors of the successive policies of policy iteration, on log–log axes. Right: the iterations needed to reach \(10^{-6}\) against \(1/(1-\gamma)\) for \(\gamma=0.9\), 0.95, 0.98, 0.99, 0.995, and 0.999, counting policy evaluations for policy iteration. The script prints the counts.

rl-dp-gambler.png

No randomness. The gambler’s problem with probability of heads 0.4 and goal 100: value 1 at capital 100 and 0 at capital 0, stakes from 1 to \(\min(s,100-s)\). In-place value iteration, sweeping capital from 1 to 99, until the largest change in a sweep is below \(10^{-12}\). Left: the value estimates after sweeps 1, 2, and 3 and at convergence. Right: every stake whose expected value is within \(10^{-9}\) of the best (red dots), over bold play, \(\min(s,100-s)\), as a thick light gray line. The script prints the number of sweeps and of optimal capital–stake pairs.

rl-dp-convergence.png

A random MDP with 200 states and 4 actions (NumPy seed 0): each state–action pair moves to 5 distinct states chosen uniformly, with probabilities from a flat Dirichlet distribution, and has a reward uniform on \([0,1]\). \(v_*\) comes from policy iteration with exact evaluation by a linear solve, starting from the first action in every state and stopping when the greedy policy repeats. Value iteration, Gauss–Seidel value iteration (in place, states in order), and modified policy iteration (a greedy step followed by 10 evaluation sweeps) start from \(v=0\) and run until the relative error \(\lVert v-v_*\rVert_\infty/\lVert v_*\rVert_\infty\) is below \(10^{-6}\). Left: the relative error at each iteration for \(\gamma=0.99\), with the errors of the successive policies of policy iteration, on log–log axes. Right: the iterations needed to reach \(10^{-6}\) against \(1/(1-\gamma)\) for \(\gamma=0.9\), 0.95, 0.98, 0.99, 0.995, and 0.999, counting policy evaluations for policy iteration. The script prints the counts.

3. Multi-armed bandits {#3-multi-armed-bandits}

Script: ch03_bandits.py.

Local figure	Data and construction
rl-bandit-testbed.png	2,000 ten-armed problems with arm means drawn from \(\mathcal N(0,1)\) and rewards equal to the mean plus \(\mathcal N(0,1)\) noise, 1,000 steps each; all draws, for the problems and then for the six agents in turn, come from one generator (NumPy seed 0). Agents: greedy and ε-greedy with \(\varepsilon=0.1\) and 0.01 on sample averages from 0; greedy with optimistic initial estimates of 5 and constant step size 0.1; UCB with \(c=2\) on sample averages, trying each untried arm first; and the gradient bandit with step size 0.1 and the average reward as baseline. Ties are broken at random. Left: the average reward over problems, smoothed by a 10-step moving average, with the mean of the best arms dotted. Right: the percentage of problems in which the best arm was chosen, smoothed the same way.
rl-bandit-regret.png	Left: five Bernoulli arms with means 0.5, 0.45, 0.4, 0.3, and 0.2, and 200 runs of 20,000 pulls for each algorithm (NumPy seed 1 for each). Each arm is pulled once first; then ε-greedy with \(\varepsilon=0.1\), UCB1 with bonus \(\sqrt{2\ln t/N}\), KL-UCB with the largest \(q\) that satisfies \(N\,\mathrm{KL}(\hat\mu,q)\le\ln t\), found by 25 bisection steps, and Thompson sampling with Beta(1, 1) priors. The curves are cumulative sums of the gaps of the chosen arms, averaged over runs, on a logarithmic time axis; the red dashed line is the Lai–Robbins rate \(\sum_a\Delta_a\ln t/\mathrm{KL}(\mu_a,\mu^*)\). Right: two arms with means 0.5 and \(0.5-\Delta\) for 11 gaps from 0.005 to 0.4, with UCB1 and Thompson sampling; the regret after \(T=2000\) pulls averaged over 400 runs (NumPy seed 2 for each gap and algorithm), with the regret \(\Delta T\) of always pulling the worse arm dotted, a vertical line at \(\Delta=\sqrt{\ln T/T}\), and the axis cut at 60. The script prints the regrets of the right panel.

rl-bandit-testbed.png

2,000 ten-armed problems with arm means drawn from \(\mathcal N(0,1)\) and rewards equal to the mean plus \(\mathcal N(0,1)\) noise, 1,000 steps each; all draws, for the problems and then for the six agents in turn, come from one generator (NumPy seed 0). Agents: greedy and ε-greedy with \(\varepsilon=0.1\) and 0.01 on sample averages from 0; greedy with optimistic initial estimates of 5 and constant step size 0.1; UCB with \(c=2\) on sample averages, trying each untried arm first; and the gradient bandit with step size 0.1 and the average reward as baseline. Ties are broken at random. Left: the average reward over problems, smoothed by a 10-step moving average, with the mean of the best arms dotted. Right: the percentage of problems in which the best arm was chosen, smoothed the same way.

rl-bandit-regret.png

Left: five Bernoulli arms with means 0.5, 0.45, 0.4, 0.3, and 0.2, and 200 runs of 20,000 pulls for each algorithm (NumPy seed 1 for each). Each arm is pulled once first; then ε-greedy with \(\varepsilon=0.1\), UCB1 with bonus \(\sqrt{2\ln t/N}\), KL-UCB with the largest \(q\) that satisfies \(N\,\mathrm{KL}(\hat\mu,q)\le\ln t\), found by 25 bisection steps, and Thompson sampling with Beta(1, 1) priors. The curves are cumulative sums of the gaps of the chosen arms, averaged over runs, on a logarithmic time axis; the red dashed line is the Lai–Robbins rate \(\sum_a\Delta_a\ln t/\mathrm{KL}(\mu_a,\mu^*)\). Right: two arms with means 0.5 and \(0.5-\Delta\) for 11 gaps from 0.005 to 0.4, with UCB1 and Thompson sampling; the regret after \(T=2000\) pulls averaged over 400 runs (NumPy seed 2 for each gap and algorithm), with the regret \(\Delta T\) of always pulling the worse arm dotted, a vertical line at \(\Delta=\sqrt{\ln T/T}\), and the axis cut at 60. The script prints the regrets of the right panel.

4. Contextual, Bayesian, and adversarial bandits {#4-contextual-bayesian-and-adversarial-bandits}

Script: ch04_contextual.py.

Local figure	Data and construction
rl-bandit-gittins.png	No randomness. The Gittins index of a Bernoulli arm with a Beta(\(a\), \(b\)) posterior and discount \(\gamma\), as the retirement reward per step \(\lambda\) at which continuing and retiring are equally good, found by 30 bisection steps between the posterior mean and 1. The value of continuing comes from backward induction over the posteriors after up to \(\lceil\ln10^{-7}/\ln\gamma\rceil\) further pulls, beyond which the arm is valued at the larger of retiring and receiving its posterior mean forever. Left: the index minus 0.5 for \(a=b=n/2\) with \(n=2\), 4, 8, …, 128 and \(\gamma=0.8\), 0.9, 0.95, and 0.99, on log–log axes, with the reference line \(0.25/\sqrt n\) of slope \(-1/2\). Right: the index against the posterior mean, for 19 means from 0.05 to 0.95 with \(a+b=10\) and \(\gamma=0.5\), 0.9, and 0.99, with the posterior mean dotted. The script prints the bonuses at \(n=2\) and 128.
rl-bandit-contextual.png	Left: 50 linear contextual problems with five actions and five-dimensional contexts, each action’s coefficients drawn from \(\mathcal N(0,I/5)\) (NumPy seed 0), standard normal contexts, and \(\mathcal N(0,0.5^2)\) reward noise; 3,000 rounds for each algorithm (NumPy seed 1 for each). LinUCB (width \(\sqrt{x^\top A_a^{-1}x}\)), linear Thompson sampling (coefficients drawn from \(\mathcal N(\hat\theta_a,0.25\,A_a^{-1})\)), and ε-greedy with \(\varepsilon=0.05\) use per-action ridge regression with \(A_a\) starting at the identity and updated by Sherman–Morrison; UCB1 ignores the context (bonus \(\sqrt{2\ln(t+1)/N}\), each action once first). Cumulative expected regret against the best action for each context, averaged over problems, on a logarithmic axis from 1 to 5,000. Right: the logged-data problem of the chapter’s code, from a new generator (NumPy seed 0): action coefficients \(w_a\) from \(\mathcal N(0,I/5)\); mean reward \(x^\top w_a+c_a\tanh(3x_1x_2)\) with \(c=(1,-1,0.5,-0.5,0)\); a logging policy that mixes, with weights 0.8 and 0.2, the softmax of \(2x^\top\tilde w_a\), where \(\tilde w_a\) is \(w_a\) in reverse order, and the uniform policy; and a target policy greedy for \(x^\top w_a\), whose true value is averaged over 200,000 contexts. 500 logged datasets of 1,000 rounds with \(\mathcal N(0,0.5^2)\) reward noise give the direct method (per-action linear regression with intercept and ridge \(10^{-3}\)), inverse propensity scoring, self-normalized IPS, and the doubly robust estimate, drawn as box plots without outliers against the true value (red dashed). The script prints each estimator’s bias and the true value.

rl-bandit-gittins.png

No randomness. The Gittins index of a Bernoulli arm with a Beta(\(a\), \(b\)) posterior and discount \(\gamma\), as the retirement reward per step \(\lambda\) at which continuing and retiring are equally good, found by 30 bisection steps between the posterior mean and 1. The value of continuing comes from backward induction over the posteriors after up to \(\lceil\ln10^{-7}/\ln\gamma\rceil\) further pulls, beyond which the arm is valued at the larger of retiring and receiving its posterior mean forever. Left: the index minus 0.5 for \(a=b=n/2\) with \(n=2\), 4, 8, …, 128 and \(\gamma=0.8\), 0.9, 0.95, and 0.99, on log–log axes, with the reference line \(0.25/\sqrt n\) of slope \(-1/2\). Right: the index against the posterior mean, for 19 means from 0.05 to 0.95 with \(a+b=10\) and \(\gamma=0.5\), 0.9, and 0.99, with the posterior mean dotted. The script prints the bonuses at \(n=2\) and 128.

rl-bandit-contextual.png

Left: 50 linear contextual problems with five actions and five-dimensional contexts, each action’s coefficients drawn from \(\mathcal N(0,I/5)\) (NumPy seed 0), standard normal contexts, and \(\mathcal N(0,0.5^2)\) reward noise; 3,000 rounds for each algorithm (NumPy seed 1 for each). LinUCB (width \(\sqrt{x^\top A_a^{-1}x}\)), linear Thompson sampling (coefficients drawn from \(\mathcal N(\hat\theta_a,0.25\,A_a^{-1})\)), and ε-greedy with \(\varepsilon=0.05\) use per-action ridge regression with \(A_a\) starting at the identity and updated by Sherman–Morrison; UCB1 ignores the context (bonus \(\sqrt{2\ln(t+1)/N}\), each action once first). Cumulative expected regret against the best action for each context, averaged over problems, on a logarithmic axis from 1 to 5,000. Right: the logged-data problem of the chapter’s code, from a new generator (NumPy seed 0): action coefficients \(w_a\) from \(\mathcal N(0,I/5)\); mean reward \(x^\top w_a+c_a\tanh(3x_1x_2)\) with \(c=(1,-1,0.5,-0.5,0)\); a logging policy that mixes, with weights 0.8 and 0.2, the softmax of \(2x^\top\tilde w_a\), where \(\tilde w_a\) is \(w_a\) in reverse order, and the uniform policy; and a target policy greedy for \(x^\top w_a\), whose true value is averaged over 200,000 contexts. 500 logged datasets of 1,000 rounds with \(\mathcal N(0,0.5^2)\) reward noise give the direct method (per-action linear regression with intercept and ridge \(10^{-3}\)), inverse propensity scoring, self-normalized IPS, and the doubly robust estimate, drawn as box plots without outliers against the true value (red dashed). The script prints each estimator’s bias and the true value.

5. Monte Carlo methods {#5-monte-carlo-methods}

Script: ch05_montecarlo.py. Both figures use blackjack with an infinite deck: cards drawn uniformly from the 13 ranks, face cards counting 10, and an ace counting 11 when that does not bust; the player hits automatically below 12, the dealer draws until 17 or more, and a hand pays \(+1\), 0, or \(-1\), with no special treatment of naturals. All blackjack hands come from one generator (NumPy seed 0), the importance-sampling runs continuing after the 500,000 hands of the first figure.

Local figure	Data and construction
rl-mc-blackjack.png	500,000 hands of the policy that sticks on 20 or 21. The estimate of each state (player sum 12–21, usable ace, dealer’s showing card) is the average return of the hands that visit it; no state repeats within a hand. Left and middle: the estimates after 10,000 and 500,000 hands for states with (top) and without (bottom) a usable ace, in the RdBu colormap from \(-1\) to 1. Right: the optimal policy (S stick, H hit), computed exactly by dynamic programming on the infinite-deck model: the distribution of the dealer’s final total from the showing card by recursion, and the values of sticking and hitting by backward induction from 21 down to 12, with ties going to sticking.
rl-mc-importance.png	Left: the state with player sum 13, a usable ace, and the dealer showing 2 (the dealer’s other card drawn), whose value under the target policy, which sticks on 20 or 21, is set to \(-0.27720\) in the script; the behavior policy sticks or hits with probability 1/2, so each step’s ratio \(\pi/b\) is 0 or 2. 100 runs of 10,000 episodes; the mean squared error of ordinary and of weighted importance sampling (0 while the weights sum to zero) against the number of episodes, on log–log axes. Right: the one-state example of infinite variance, in which “left” returns to the state with probability 0.9 or ends with \(+1\) and “right” ends with 0; the target always goes left (value 1) and the behavior picks each action with probability 1/2. Ten runs of \(10^6\) episodes (NumPy seed 5) are sampled in closed form: the number \(k\) of left-and-return steps is geometric with success probability 0.55, minus 1; the episode ends by “left” with \(+1\) with probability \(0.05/0.55\), and then its ratio is \(2^{k+1}\), otherwise 0. The ordinary importance-sampling estimate at 400 logarithmically spaced episode counts, with the axis cut at 3 and the true value dashed. The script prints each run’s final estimate.

rl-mc-blackjack.png

500,000 hands of the policy that sticks on 20 or 21. The estimate of each state (player sum 12–21, usable ace, dealer’s showing card) is the average return of the hands that visit it; no state repeats within a hand. Left and middle: the estimates after 10,000 and 500,000 hands for states with (top) and without (bottom) a usable ace, in the RdBu colormap from \(-1\) to 1. Right: the optimal policy (S stick, H hit), computed exactly by dynamic programming on the infinite-deck model: the distribution of the dealer’s final total from the showing card by recursion, and the values of sticking and hitting by backward induction from 21 down to 12, with ties going to sticking.

rl-mc-importance.png

Left: the state with player sum 13, a usable ace, and the dealer showing 2 (the dealer’s other card drawn), whose value under the target policy, which sticks on 20 or 21, is set to \(-0.27720\) in the script; the behavior policy sticks or hits with probability 1/2, so each step’s ratio \(\pi/b\) is 0 or 2. 100 runs of 10,000 episodes; the mean squared error of ordinary and of weighted importance sampling (0 while the weights sum to zero) against the number of episodes, on log–log axes. Right: the one-state example of infinite variance, in which “left” returns to the state with probability 0.9 or ends with \(+1\) and “right” ends with 0; the target always goes left (value 1) and the behavior picks each action with probability 1/2. Ten runs of \(10^6\) episodes (NumPy seed 5) are sampled in closed form: the number \(k\) of left-and-return steps is geometric with success probability 0.55, minus 1; the episode ends by “left” with \(+1\) with probability \(0.05/0.55\), and then its ratio is \(2^{k+1}\), otherwise 0. The ordinary importance-sampling estimate at 400 logarithmically spaced episode counts, with the axis cut at 3 and the true value dashed. The script prints each run’s final estimate.

6. Temporal-difference learning {#6-temporal-difference-learning}

Script: ch06_td.py.

Local figure	Data and construction
rl-td-randomwalk.png	The five-state random walk: start in the middle state, move left or right with probability 1/2, reward 1 on leaving to the right and 0 on leaving to the left, undiscounted, true values \(1/6,\dots,5/6\); estimates start at 0.5, and all walks come from one generator (NumPy seed 0). Left: TD(0) with \(\alpha=0.05\), 0.1, and 0.15 and constant-α every-visit Monte Carlo with \(\alpha=0.01\), 0.02, 0.03, and 0.04, each with 100 runs of 100 episodes; the RMS error over the five states after each episode, averaged over runs. Right: batch updating on 100 runs of 100 episodes, using the first \(n\) episodes for \(n=1,\dots,100\): batch Monte Carlo as the sample averages of the returns (0.5 for unvisited states), and batch TD as the certainty-equivalence solution of the Markov reward process estimated from the transition counts, computed by a linear solve. The script prints the batch errors after 10 and 100 episodes.
rl-td-dopamine.png	No randomness. Trials of 50 steps with a cue at step 10 and a reward of 1 at step 30. The steps before the cue have value 0 and are not learned; each of the 40 steps from the cue on has its own value (a tapped delay line), learned from 0 by TD(0) with \(\gamma=0.98\) and \(\alpha=0.1\). Left: the TD errors during trials 1, 10, and 30. Middle: the errors of a trial without updates after 500 learning trials. Right: the same trial with the reward omitted. Dotted lines mark the cue and the reward time.

rl-td-randomwalk.png

The five-state random walk: start in the middle state, move left or right with probability 1/2, reward 1 on leaving to the right and 0 on leaving to the left, undiscounted, true values \(1/6,\dots,5/6\); estimates start at 0.5, and all walks come from one generator (NumPy seed 0). Left: TD(0) with \(\alpha=0.05\), 0.1, and 0.15 and constant-α every-visit Monte Carlo with \(\alpha=0.01\), 0.02, 0.03, and 0.04, each with 100 runs of 100 episodes; the RMS error over the five states after each episode, averaged over runs. Right: batch updating on 100 runs of 100 episodes, using the first \(n\) episodes for \(n=1,\dots,100\): batch Monte Carlo as the sample averages of the returns (0.5 for unvisited states), and batch TD as the certainty-equivalence solution of the Markov reward process estimated from the transition counts, computed by a linear solve. The script prints the batch errors after 10 and 100 episodes.

rl-td-dopamine.png

No randomness. Trials of 50 steps with a cue at step 10 and a reward of 1 at step 30. The steps before the cue have value 0 and are not learned; each of the 40 steps from the cue on has its own value (a tapped delay line), learned from 0 by TD(0) with \(\gamma=0.98\) and \(\alpha=0.1\). Left: the TD errors during trials 1, 10, and 30. Middle: the errors of a trial without updates after 500 learning trials. Right: the same trial with the reward omitted. Dotted lines mark the cue and the reward time.

7. Model-free control {#7-model-free-control}

Script: ch07_control.py.

Local figure	Data and construction
rl-control-cliff.png	Cliff walking: a \(4\times12\) grid with start and goal at the two ends of the bottom row, reward \(-1\) per step, and \(-100\) and a return to the start for stepping into the cliff, the rest of the bottom row; moves off the grid leave the agent in place; undiscounted. SARSA, Q-learning, and Expected SARSA with ε-greedy behavior (\(\varepsilon=0.1\), ties broken at random) and action values from 0, with a generator for each run. Top left: \(\alpha=0.5\) and 500 episodes; the sum of rewards per episode, averaged over 50 runs (NumPy seeds 0–49) and smoothed by a 10-episode moving average, with the axis cut at \(-100\). Top right: the average return over the first 100 episodes for \(\alpha=0.1\), 0.2, …, 1, averaged over 20 runs (NumPy seeds 0–19). Bottom: the greedy routes, at most 60 steps, from the action values of the run with seed 0 after 500 episodes with \(\alpha=0.5\), with their lengths in the legend. The script prints the returns of the top-right panel.
rl-control-maxbias.png	The maximization-bias MDP: from A, “right” ends the episode with reward 0 and “left” leads to B with reward 0, where each of 10 actions ends it with reward drawn from \(\mathcal N(-0.1,1)\). Q-learning and double Q-learning (each update chooses one of the two tables with probability 1/2 and evaluates its greedy action with the other; the behavior is ε-greedy on their sum), with \(\alpha=0.1\), \(\varepsilon=0.1\), no discounting, ties broken at random, and 2,000 runs of 300 episodes, all draws, for both methods in turn, from one generator (NumPy seed 0). The percentage of runs that go left from A in each episode, with the 5% of an agent that prefers right and explores with \(\varepsilon=0.1\) dotted.

rl-control-cliff.png

Cliff walking: a \(4\times12\) grid with start and goal at the two ends of the bottom row, reward \(-1\) per step, and \(-100\) and a return to the start for stepping into the cliff, the rest of the bottom row; moves off the grid leave the agent in place; undiscounted. SARSA, Q-learning, and Expected SARSA with ε-greedy behavior (\(\varepsilon=0.1\), ties broken at random) and action values from 0, with a generator for each run. Top left: \(\alpha=0.5\) and 500 episodes; the sum of rewards per episode, averaged over 50 runs (NumPy seeds 0–49) and smoothed by a 10-episode moving average, with the axis cut at \(-100\). Top right: the average return over the first 100 episodes for \(\alpha=0.1\), 0.2, …, 1, averaged over 20 runs (NumPy seeds 0–19). Bottom: the greedy routes, at most 60 steps, from the action values of the run with seed 0 after 500 episodes with \(\alpha=0.5\), with their lengths in the legend. The script prints the returns of the top-right panel.

rl-control-maxbias.png

The maximization-bias MDP: from A, “right” ends the episode with reward 0 and “left” leads to B with reward 0, where each of 10 actions ends it with reward drawn from \(\mathcal N(-0.1,1)\). Q-learning and double Q-learning (each update chooses one of the two tables with probability 1/2 and evaluates its greedy action with the other; the behavior is ε-greedy on their sum), with \(\alpha=0.1\), \(\varepsilon=0.1\), no discounting, ties broken at random, and 2,000 runs of 300 episodes, all draws, for both methods in turn, from one generator (NumPy seed 0). The percentage of runs that go left from A in each episode, with the 5% of an agent that prefers right and explores with \(\varepsilon=0.1\) dotted.

8. Multi-step bootstrapping and eligibility traces {#8-multi-step-bootstrapping-and-eligibility-traces}

Script: ch08_traces.py. The first two figures use the 19-state random walk: start in the middle, reward \(-1\) on leaving to the left and \(+1\) on leaving to the right, undiscounted, true values \(-0.9,\dots,0.9\), and estimates from 0. 100 runs of 10 episodes are drawn once (NumPy seed 0) and shared by every method and step size; the step sizes run from 0.05 to 1 in steps of 0.05, and the error is the RMS error over the 19 states after each episode, averaged over the 10 episodes and the 100 runs.

Local figure	Data and construction
rl-traces-nstep.png	\(n\)-step TD for \(n=1\), 2, 4, …, 512, colored by the viridis colormap, on an axis limited to 0.25–0.55. The script prints each \(n\)’s lowest error and its step size.
rl-traces-lambda.png	The offline λ-return algorithm (λ-returns computed at the end of each episode from the estimates at its start, then applied in order), TD(λ) with accumulating traces, and true online TD(λ) with dutch traces, for \(\lambda=0\), 0.4, 0.8, 0.9, 0.95, 0.975, 0.99, and 1. Estimates that stop being finite are replaced by \(\pm1000\) (NaN by 1000), and errors are drawn up to 0.6 on an axis limited to 0.25–0.55. The script prints each method’s lowest error and its step size for each λ.
rl-traces-concept.png	No randomness. Left: the weights \((1-\lambda)\lambda^{n-1}\) of the \(n\)-step returns, \(n=1,\dots,11\), and the remaining weight \(\lambda^{11}\) on the Monte Carlo return, for a state 12 steps before the end of the episode, with \(\lambda=0.5\), 0.8, and 0.95. Right: the accumulating, replacing, and dutch (\(\alpha=0.5\)) traces of a state visited at steps 0, 4, 5, 6, and 15, over 25 steps with \(\gamma\lambda=0.8\).

9. Off-policy learning {#9-off-policy-learning}

Script: ch09_offpolicy.py. Both figures use random MDPs with 10 states and 3 actions whose transition rows are drawn from a Dirichlet distribution with parameters 0.5; each figure builds its MDP from a new generator (NumPy seed 0).

Local figure	Data and construction
rl-offpolicy-horizon.png	Mean rewards from \(\mathcal N(0,1)\) for each state–action pair, observed with \(\mathcal N(0,1)\) noise; a target policy that is the softmax of standard normal logits in each state; and the behavior \(\mu=\tfrac12\pi+\tfrac16\). A model for the direct and doubly robust estimators is estimated from one behavior trajectory of 500 steps that restarts at state 0 every 50 steps, with transition counts plus 0.1 and mean observed rewards. For horizons \(H=5\), 10, 20, 30, 50, 75, and 100, with undiscounted episodes from state 0, 1,000 datasets of 100 episodes give ordinary and weighted importance sampling over trajectories, per-decision importance sampling, the direct method (the model’s value, the same for every dataset), the per-decision doubly robust estimator, and marginalized importance sampling with the exact ratios of the state distributions at each step; the true value comes from backward induction. The RMS error over datasets divided by the true value, on a logarithmic axis from 0.03 to 5. The script prints the errors for each horizon.
rl-offpolicy-traces.png	Rewards \(r(s,a)\) drawn from \(\mathcal N(0,1)\) and received without noise; a target policy that is the softmax of twice standard normal logits; a second policy \(\nu\) with rows from a flat Dirichlet distribution; the behavior \(\mu=(1-\beta)\pi+\beta\nu\) for \(\beta=0\), 0.5, and 1; \(\gamma=0.9\); and \(q_\pi\) by a linear solve. For each β in turn, the generator continuing, 20 runs in parallel for 100,000 steps, with the traces cleared and a uniformly drawn state every 50 steps. Updates with \(\lambda=1\), \(\alpha=0.01\), the TD error \(r+\gamma\sum_{a'}\pi(a'\mid s')Q(s',a')-Q(s,a)\), and accumulating traces decayed by \(\gamma\lambda c\), where \(c\) is \(\pi/\mu\) (importance sampling), \(\pi\) (tree backup), 1 (\(Q^\pi(\lambda)\)), or \(\min(1,\pi/\mu)\) (Retrace) at the next pair. Every 2,500 steps, the RMS error over states of \(\sum_a\pi(a\mid s)Q(s,a)\) against \(v_\pi\), the median over runs (errors that are not finite counted as \(10^6\)), on a logarithmic axis from 0.03 to 10. The script prints the final errors.

rl-offpolicy-horizon.png

Mean rewards from \(\mathcal N(0,1)\) for each state–action pair, observed with \(\mathcal N(0,1)\) noise; a target policy that is the softmax of standard normal logits in each state; and the behavior \(\mu=\tfrac12\pi+\tfrac16\). A model for the direct and doubly robust estimators is estimated from one behavior trajectory of 500 steps that restarts at state 0 every 50 steps, with transition counts plus 0.1 and mean observed rewards. For horizons \(H=5\), 10, 20, 30, 50, 75, and 100, with undiscounted episodes from state 0, 1,000 datasets of 100 episodes give ordinary and weighted importance sampling over trajectories, per-decision importance sampling, the direct method (the model’s value, the same for every dataset), the per-decision doubly robust estimator, and marginalized importance sampling with the exact ratios of the state distributions at each step; the true value comes from backward induction. The RMS error over datasets divided by the true value, on a logarithmic axis from 0.03 to 5. The script prints the errors for each horizon.

rl-offpolicy-traces.png

Rewards \(r(s,a)\) drawn from \(\mathcal N(0,1)\) and received without noise; a target policy that is the softmax of twice standard normal logits; a second policy \(\nu\) with rows from a flat Dirichlet distribution; the behavior \(\mu=(1-\beta)\pi+\beta\nu\) for \(\beta=0\), 0.5, and 1; \(\gamma=0.9\); and \(q_\pi\) by a linear solve. For each β in turn, the generator continuing, 20 runs in parallel for 100,000 steps, with the traces cleared and a uniformly drawn state every 50 steps. Updates with \(\lambda=1\), \(\alpha=0.01\), the TD error \(r+\gamma\sum_{a'}\pi(a'\mid s')Q(s',a')-Q(s,a)\), and accumulating traces decayed by \(\gamma\lambda c\), where \(c\) is \(\pi/\mu\) (importance sampling), \(\pi\) (tree backup), 1 (\(Q^\pi(\lambda)\)), or \(\min(1,\pi/\mu)\) (Retrace) at the next pair. Every 2,500 steps, the RMS error over states of \(\sum_a\pi(a\mid s)Q(s,a)\) against \(v_\pi\), the median over runs (errors that are not finite counted as \(10^6\)), on a logarithmic axis from 0.03 to 10. The script prints the final errors.

10. Planning and learning with tabular models {#10-planning-and-learning-with-tabular-models}

Script: ch10_planning.py.

Local figure	Data and construction
rl-planning-dyna.png	A \(6\times9\) maze (rows counted from the top, from 0) with walls in rows 1–3 of column 2, row 4 of column 5, and rows 0–2 of column 7, the start at row 2 of column 0, the goal in the top-right corner with reward 1, and \(\gamma=0.95\). Dyna-Q with step size 0.1, ε-greedy behavior (\(\varepsilon=0.1\), ties broken at random), a deterministic model that stores the last outcome of each state–action pair, and 0, 5, or 50 planning updates per step on pairs drawn uniformly, with replacement, from those observed; 50 episodes and 30 runs (NumPy seeds 0–29). Left: steps per episode from episode 2 on, averaged over runs, on a logarithmic axis, with the shortest path, 14 steps, dotted. Middle and right: the greedy actions after two episodes of the run with seed 0, for 0 and 50 planning steps, in the states whose largest action value is positive. The script prints the steps of episode 2 and the mean of episodes 41–50.
rl-planning-changing.png	The same grid and goal, with the start at row 5, column 3 and a wall along row 3. Blocking maze: the wall covers columns 0–7 for the first 1,000 steps and columns 1–8 afterward; shortcut maze: columns 1–8 for the first 3,000 steps and columns 1–7 afterward. Dyna-Q and Dyna-Q+ with step size 1, 10 planning updates per step, \(\gamma=0.95\), and \(\varepsilon=0.1\); Dyna-Q+ adds \(\kappa\sqrt\tau\) with \(\kappa=10^{-3}\) to the modeled reward of a pair last tried \(\tau\) steps earlier, and on its first visit to a state enters the untried actions in the model as leading back to the state with reward 0. The agent restarts after reaching the goal. The cumulative reward over 3,000 and 6,000 steps, averaged over 20 runs (NumPy seeds 0–19), with the change dotted.
rl-planning-updates.png	Left: for branching factors \(b=2\), 10, 100, 1,000, and 10,000, 200 runs of \(b\) successor values drawn from \(\mathcal N(0,1)\) and standardized within each run to mean 0 and standard deviation 1 (NumPy seed 0, one generator for all \(b\)); sample updates with step size \(1/t\) average \(2b\) successors drawn uniformly with replacement. The RMS error over runs against the true value, 0, as a function of \(t/b\), with the expected update dashed (error 1 until \(b\) evaluations, then 0). Right: random tasks with 1,000 states, 2 actions, and \(b=1\), 3, or 10 successors per pair drawn uniformly, each with its own \(\mathcal N(0,1)\) reward; the expected update sets \(Q(s,a)\) to the mean over its successors of the reward plus \(0.9\max_{a'}Q(s',a')\). Uniform: the pairs in a fixed cycle. On-policy: the pairs of an ε-greedy (\(\varepsilon=0.1\)) trajectory from state 0 that returns to state 0 with probability 0.1 at each step. 20,000 updates; every 500, the value of state 0 under the greedy policy by 100 sweeps of evaluation with discount 0.9, averaged over 30 tasks (NumPy seeds 0–29, the same tasks for both methods). The script prints some of these values.

11. Value function approximation {#11-value-function-approximation}

Script: ch11_fa.py. The first two figures use the 1000-state random walk: from each state, a jump to one of the 100 states on either side, each with probability 1/200; a jump past the left end ends the episode with \(-1\) and past the right end with \(+1\); undiscounted, with the start at state 500. The true values come from a linear solve, and the on-policy distribution \(\mu\) is the normalized expected number of visits from the start.

Local figure	Data and construction
rl-fa-aggregation.png	No randomness. Ten groups of 100 states. The limit of gradient Monte Carlo is the \(\mu\)-weighted least-squares projection of the true values, and the semi-gradient TD fixed point is solved from \(X^\top D(I-P)Xw=X^\top Dr\) with \(D=\operatorname{diag}(\mu)\). Both staircases are drawn over the true values, with \(\mu\) shaded on a second axis cut at 0.006, which clips the start state.
rl-fa-features.png	Top left: \(\cos(i\pi s)\) for \(i=1,\dots,4\) on \([0,1]\). Bottom left: a schematic of four tilings of the unit interval with tiles of width 0.25, each offset by a quarter of a tile from the last, and one state’s active tiles. Right: gradient Monte Carlo with polynomial features \(s^0,\dots,s^n\) (step size \(10^{-4}\)) and Fourier features \(\cos(i\pi s)\), \(i=0,\dots,n\) (step size \(5\times10^{-5}\)), of orders \(n=5\), 10, and 20, on the states scaled to \([0,1]\); 10 runs of 5,000 episodes (NumPy seed 0, one generator), with all six bases learning from the same episodes in each run. Each episode makes one update, the sum of \(\alpha\,x_t(G-x_t^\top w)\) over its visited states with the weights at its start. The \(\mu\)-weighted RMS value error every 50 episodes, averaged over runs. The script prints the final errors.
rl-fa-mountaincar.png	Mountain car: \(v\leftarrow v+0.001(a-1)-0.0025\cos3x\) clipped to \([-0.07,0.07]\), \(x\leftarrow x+v\) clipped to \([-1.2,0.6]\), the velocity set to 0 at the left bound, the goal at \(x\ge0.5\), reward \(-1\) per step, starts uniform on \([-0.6,-0.4]\) at rest, and episodes capped at 10,000 steps. Tile coding with 8 tilings of \(9\times9\) tiles over \([-1.2,0.5]\times[-0.07,0.07]\), tiling \(t\) offset by \(t/8\) of a tile in position and \((3t\bmod8)/8\) in velocity, with separate weights for the three actions, starting at 0. Semi-gradient SARSA, undiscounted, acting greedily with ties broken at random. Left: after 1,000 episodes with \(\alpha=0.5/8\) (NumPy seed 0), the cost to go \(-\max_a\hat q\) on a \(120\times120\) grid (viridis colormap), and the greedy trajectory from rest at \(-0.5\) (at most 1,000 steps). Right: steps per episode for \(\alpha=0.1/8\), 0.2/8, and 0.5/8, averaged over 10 runs of 500 episodes (NumPy seeds 0–9) and smoothed by a 10-episode moving average, on a logarithmic axis. The script prints the mean of the last 10 episodes for each step size.

12. The deadly triad and gradient-TD methods {#12-the-deadly-triad-and-gradient-td-methods}

Script: ch12_triad.py.

Local figure	Data and construction
rl-triad-baird.png	Baird’s counterexample: seven states with features \(2e_i+e_8\) for states 1–6 and \(e_7+2e_8\) for state 7, zero rewards, \(\gamma=0.99\), and \(w_0=(1,1,1,1,1,1,10,1)\); the target policy always moves to state 7, and the behavior moves there with probability 1/7 and otherwise to one of states 1–6 uniformly. Left: semi-gradient off-policy TD(0) on 1,000 sampled transitions with \(\alpha=0.01\) and importance ratios 7 or 0, from a uniformly drawn start (NumPy seed 0). Middle: semi-gradient dynamic programming, the expected update with uniform state weights, for 1,000 sweeps with \(\alpha=0.01\). Right: expected TDC (\(\alpha=0.005\), secondary step size 0.05) and expected emphatic TD(0) (\(\alpha=0.003\)) with emphasis 1/7 on states 1–6 and \((1/7+6\gamma/7)/(1-\gamma)\) on state 7, both for 1,000 sweeps; the RMS value error of both (the true values are 0) and TDC’s RMS projected Bellman error, which equals the Bellman error here, with uniform state weights and the axis cut at 10. The script prints the final errors.
rl-triad-geometry.png	No data are used. A schematic of the plane of representable value functions with \(v_\pi\), its projection \(\Pi v_\pi\), a representable \(v_{\mathbf w}\), \(B_\pi v_{\mathbf w}\), \(\Pi B_\pi v_{\mathbf w}\), and the TD fixed point, and arrows for the value error, the Bellman error, and the projected Bellman error.

rl-triad-baird.png

Baird’s counterexample: seven states with features \(2e_i+e_8\) for states 1–6 and \(e_7+2e_8\) for state 7, zero rewards, \(\gamma=0.99\), and \(w_0=(1,1,1,1,1,1,10,1)\); the target policy always moves to state 7, and the behavior moves there with probability 1/7 and otherwise to one of states 1–6 uniformly. Left: semi-gradient off-policy TD(0) on 1,000 sampled transitions with \(\alpha=0.01\) and importance ratios 7 or 0, from a uniformly drawn start (NumPy seed 0). Middle: semi-gradient dynamic programming, the expected update with uniform state weights, for 1,000 sweeps with \(\alpha=0.01\). Right: expected TDC (\(\alpha=0.005\), secondary step size 0.05) and expected emphatic TD(0) (\(\alpha=0.003\)) with emphasis 1/7 on states 1–6 and \((1/7+6\gamma/7)/(1-\gamma)\) on state 7, both for 1,000 sweeps; the RMS value error of both (the true values are 0) and TDC’s RMS projected Bellman error, which equals the Bellman error here, with uniform state weights and the axis cut at 10. The script prints the final errors.

rl-triad-geometry.png

No data are used. A schematic of the plane of representable value functions with \(v_\pi\), its projection \(\Pi v_\pi\), a representable \(v_{\mathbf w}\), \(B_\pi v_{\mathbf w}\), \(\Pi B_\pi v_{\mathbf w}\), and the TD fixed point, and arrows for the value error, the Bellman error, and the projected Bellman error.

13. Policy gradient and actor-critic methods {#13-policy-gradient-and-actor-critic-methods}

Script: ch13_pg.py.

Local figure	Data and construction
rl-pg-corridor.png	The short corridor with switched actions: three nonterminal states and a goal, reward \(-1\) per step, “left” in the first state staying put, and the effects of the actions reversed in the second state; undiscounted. The policy goes right with the same probability \(p\) in every state, a logistic function of two action preferences. Left: the exact value of the start state \(J(p)\), by a linear solve, at 400 values of \(p\) from 0.02 to 0.98, with its maximum and the ε-greedy policies \(p=0.05\) and 0.95. Middle and right: REINFORCE from \(p=0.05\), updating after each episode with the undiscounted returns to go and, where stated, a learned scalar baseline updated at every step; episodes are capped at 1,000 steps, and each setting has 100 runs of 1,000 episodes. The settings of each panel run in parallel, with all draws from one generator (NumPy seed 0), first for the middle panel and then for the right. Middle: step sizes \(2^{-13}\) and \(2^{-11}\) without baseline and \(\alpha_\theta=2^{-9}\), \(\alpha_w=2^{-6}\) with it; the median over runs of the exact value \((2p-4)/(p(1-p))\) of each run’s current policy, on a logarithmic episode axis, with the optimum dotted. Right: the percentage of runs with an episode that reached the 1,000-step cap, for \(\alpha_\theta=2^{-12},\dots,2^{-6}\), without baseline and with \(\alpha_w=2^{-6}\). The script prints the medians after 10, 100, and 1,000 episodes and the percentages.
rl-pg-natural.png	No randomness. A chain of 10 states with deterministic left and right moves (staying put at the ends), reward 1 for either action in the last state, \(\gamma=0.95\), and start state 0, whose optimal value is \(\gamma^9/(1-\gamma)\). A tabular softmax policy that goes left with probability 0.5 (solid) or 0.99 (dashed) in every state. Exact policy gradient, \(\theta\leftarrow\theta+\eta\,d(s)\,\pi(a\mid s)\,A(s,a)/(1-\gamma)\) with \(d\) the normalized discounted state distribution from state 0, and natural policy gradient, \(\theta\leftarrow\theta+\eta\,A(s,a)/(1-\gamma)\), both with \(\eta=1\) for 3,000 iterations; the value of state 0 relative to the optimum, on a logarithmic iteration axis.

rl-pg-corridor.png

The short corridor with switched actions: three nonterminal states and a goal, reward \(-1\) per step, “left” in the first state staying put, and the effects of the actions reversed in the second state; undiscounted. The policy goes right with the same probability \(p\) in every state, a logistic function of two action preferences. Left: the exact value of the start state \(J(p)\), by a linear solve, at 400 values of \(p\) from 0.02 to 0.98, with its maximum and the ε-greedy policies \(p=0.05\) and 0.95. Middle and right: REINFORCE from \(p=0.05\), updating after each episode with the undiscounted returns to go and, where stated, a learned scalar baseline updated at every step; episodes are capped at 1,000 steps, and each setting has 100 runs of 1,000 episodes. The settings of each panel run in parallel, with all draws from one generator (NumPy seed 0), first for the middle panel and then for the right. Middle: step sizes \(2^{-13}\) and \(2^{-11}\) without baseline and \(\alpha_\theta=2^{-9}\), \(\alpha_w=2^{-6}\) with it; the median over runs of the exact value \((2p-4)/(p(1-p))\) of each run’s current policy, on a logarithmic episode axis, with the optimum dotted. Right: the percentage of runs with an episode that reached the 1,000-step cap, for \(\alpha_\theta=2^{-12},\dots,2^{-6}\), without baseline and with \(\alpha_w=2^{-6}\). The script prints the medians after 10, 100, and 1,000 episodes and the percentages.

rl-pg-natural.png

No randomness. A chain of 10 states with deterministic left and right moves (staying put at the ends), reward 1 for either action in the last state, \(\gamma=0.95\), and start state 0, whose optimal value is \(\gamma^9/(1-\gamma)\). A tabular softmax policy that goes left with probability 0.5 (solid) or 0.99 (dashed) in every state. Exact policy gradient, \(\theta\leftarrow\theta+\eta\,d(s)\,\pi(a\mid s)\,A(s,a)/(1-\gamma)\) with \(d\) the normalized discounted state distribution from state 0, and natural policy gradient, \(\theta\leftarrow\theta+\eta\,A(s,a)/(1-\gamma)\), both with \(\eta=1\) for 3,000 iterations; the value of state 0 relative to the optimum, on a logarithmic iteration axis.

14. Partially observable environments {#14-partially-observable-environments}

Script: ch14_pomdp.py.

Local figure Data and construction

rl-pomdp-tiger.png No randomness. The tiger POMDP of chapter 1 (listening accuracy 0.85, rewards \(-1\), \(-100\), and 10, \(\gamma=0.95\)). Exact value iteration from the zero vector for horizons 1 to 10: the cross-sum backup of each action is pruned, and then their union is pruned again; after duplicates are removed, a vector is kept if it is maximal at one of 201 evenly spaced beliefs or if SciPy’s linprog finds a belief at which it exceeds all the others by more than \(10^{-9}\). Left: all alpha-vectors at horizon 10 (light gray) and their upper envelope, colored by the best action, on an axis from \(-40\) to 30. Right: the number of vectors at each horizon after pruning, and the count \(\lvert\mathcal A\rvert\,\lvert\Gamma_{t-1}\rvert^{\lvert\mathcal O\rvert}\) without pruning, starting from 3, up to horizon 6, on a logarithmic axis. The script prints the counts.

rl-pomdp-heaven.png Heaven and hell on a map of 11 open cells: a top corridor with an exit at each end, and a vertical corridor from its middle down past the start to the priest. The state is the cell and the side of heaven, plus a final state. Moves (up, down, left, right) are deterministic, and a move into a wall stays put; entering the heaven exit pays \(+1\) and the other exit \(-1\), after which the episode ends; \(\gamma=0.95\). The observation is the cell and, at the priest’s cell, also the side of heaven, and the start belief puts 1/2 on each side at the start cell. Point-based value iteration: 300 point-based backups over the start belief and the distinct beliefs, rounded to 6 decimals, reached by 300 random walks of 30 uniformly random actions from the start (NumPy seed 0). QMDP: 500 sweeps of value iteration on the underlying MDP, acting on the largest \(\sum_s b(s)Q(s,a)\). The deterministic paths of both policies with heaven on the left, at most 40 steps; the returns in the annotations (\(\gamma^{10}\approx0.60\) and 0) are written in the script.

Local figure	Data and construction
rl-pomdp-tiger.png	No randomness. The tiger POMDP of chapter 1 (listening accuracy 0.85, rewards \(-1\), \(-100\), and 10, \(\gamma=0.95\)). Exact value iteration from the zero vector for horizons 1 to 10: the cross-sum backup of each action is pruned, and then their union is pruned again; after duplicates are removed, a vector is kept if it is maximal at one of 201 evenly spaced beliefs or if SciPy’s `linprog` finds a belief at which it exceeds all the others by more than \(10^{-9}\). Left: all alpha-vectors at horizon 10 (light gray) and their upper envelope, colored by the best action, on an axis from \(-40\) to 30. Right: the number of vectors at each horizon after pruning, and the count \(\lvert\mathcal A\rvert\,\lvert\Gamma_{t-1}\rvert^{\lvert\mathcal O\rvert}\) without pruning, starting from 3, up to horizon 6, on a logarithmic axis. The script prints the counts.
rl-pomdp-heaven.png	Heaven and hell on a map of 11 open cells: a top corridor with an exit at each end, and a vertical corridor from its middle down past the start to the priest. The state is the cell and the side of heaven, plus a final state. Moves (up, down, left, right) are deterministic, and a move into a wall stays put; entering the heaven exit pays \(+1\) and the other exit \(-1\), after which the episode ends; \(\gamma=0.95\). The observation is the cell and, at the priest’s cell, also the side of heaven, and the start belief puts 1/2 on each side at the start cell. Point-based value iteration: 300 point-based backups over the start belief and the distinct beliefs, rounded to 6 decimals, reached by 300 random walks of 30 uniformly random actions from the start (NumPy seed 0). QMDP: 500 sweeps of value iteration on the underlying MDP, acting on the largest \(\sum_s b(s)Q(s,a)\). The deterministic paths of both policies with heaven on the left, at most 40 steps; the returns in the annotations (\(\gamma^{10}\approx0.60\) and 0) are written in the script.

15. Optimal control and trajectory optimization {#15-optimal-control-and-trajectory-optimization}

Script: ch15_control.py.

Local figure Data and construction

rl-control-lqr.png The classic cart-pole equations (gravity 9.8, cart mass 1, pole mass 0.1, half-length 0.5) with Euler steps of 0.02 s, linearized at the upright rest state by central differences with step \(10^{-6}\); costs \(Q=\operatorname{diag}(1,0.1,10,0.1)\) on position, velocity, angle, and angular velocity, and \(R=0.01\). Left (no randomness): 400 steps of the Riccati recursion from \(P=Q\), each gain divided by its value at step 400, on a logarithmic axis. Middle (no randomness): the stationary controller on the nonlinear cart-pole with the force clipped to \(\pm10\) N, from 20°, 25°, and 30° at rest, for 250 steps or until the pole passes 90°. Right: measurements of position and angle with noise of standard deviations 0.02 and 1°, process noise of standard deviations 0.02 on the velocity and 0.05 on the angular velocity, and the steady-state Kalman gain from SciPy’s solve_discrete_are. The controller acts on the true state, on the Kalman filter’s estimate (starting from zero), or on the measurements with velocities from differences, each run from 3° (NumPy seed 3 for each) for 250 steps or until the pole passes 90° or the cart leaves \([-2.4,2.4]\).

rl-control-swingup.png A pendulum with the angle measured from upright, \(\ddot\theta=15\sin\theta+3u/m\), semi-implicit Euler steps of 0.05 s, torque in \([-2,2]\), and cost \(2(1-\cos\theta)+0.1\,\omega^2+0.001\,u^2\) per step and the same without torque at the end, starting from hanging at rest; the model has \(m=1\). iLQR: a 100-step plan with torque \(2\tanh v\), from \(v\) equal to 0.1 times standard normal values (NumPy seed 0), with regularization of \(Q_{uu}\) starting at 1, halved after a successful iteration and multiplied by 10 after a failed one, a line search over step sizes 1, 0.5, 0.25, 0.1, 0.05, and 0.01, and at most 300 iterations, stopping when the relative improvement falls below \(10^{-6}\). MPC with MPPI: at each of 150 steps, 3 iterations over 300 perturbed 50-step torque sequences (noise standard deviation 1, clipped to \([-2,2]\), temperature 0.3), warm-started by shifting the previous plan (NumPy seed 0). Left: the height of the tip, \(\cos\theta\), for both. Middle: their torques. Right: a pendulum with \(m=1.2\) and a torque disturbance of 0.1 times a standard normal value at each step (NumPy seed 1 for each controller): the MPPI torques replayed open loop, the MPPI trajectory tracked by time-varying LQR (\(Q=\operatorname{diag}(1,0.1)\), \(R=0.01\), linearized along it), and MPC with MPPI on the nominal model.

Local figure	Data and construction
rl-control-lqr.png	The classic cart-pole equations (gravity 9.8, cart mass 1, pole mass 0.1, half-length 0.5) with Euler steps of 0.02 s, linearized at the upright rest state by central differences with step \(10^{-6}\); costs \(Q=\operatorname{diag}(1,0.1,10,0.1)\) on position, velocity, angle, and angular velocity, and \(R=0.01\). Left (no randomness): 400 steps of the Riccati recursion from \(P=Q\), each gain divided by its value at step 400, on a logarithmic axis. Middle (no randomness): the stationary controller on the nonlinear cart-pole with the force clipped to \(\pm10\) N, from 20°, 25°, and 30° at rest, for 250 steps or until the pole passes 90°. Right: measurements of position and angle with noise of standard deviations 0.02 and 1°, process noise of standard deviations 0.02 on the velocity and 0.05 on the angular velocity, and the steady-state Kalman gain from SciPy’s `solve_discrete_are`. The controller acts on the true state, on the Kalman filter’s estimate (starting from zero), or on the measurements with velocities from differences, each run from 3° (NumPy seed 3 for each) for 250 steps or until the pole passes 90° or the cart leaves \([-2.4,2.4]\).
rl-control-swingup.png	A pendulum with the angle measured from upright, \(\ddot\theta=15\sin\theta+3u/m\), semi-implicit Euler steps of 0.05 s, torque in \([-2,2]\), and cost \(2(1-\cos\theta)+0.1\,\omega^2+0.001\,u^2\) per step and the same without torque at the end, starting from hanging at rest; the model has \(m=1\). iLQR: a 100-step plan with torque \(2\tanh v\), from \(v\) equal to 0.1 times standard normal values (NumPy seed 0), with regularization of \(Q_{uu}\) starting at 1, halved after a successful iteration and multiplied by 10 after a failed one, a line search over step sizes 1, 0.5, 0.25, 0.1, 0.05, and 0.01, and at most 300 iterations, stopping when the relative improvement falls below \(10^{-6}\). MPC with MPPI: at each of 150 steps, 3 iterations over 300 perturbed 50-step torque sequences (noise standard deviation 1, clipped to \([-2,2]\), temperature 0.3), warm-started by shifting the previous plan (NumPy seed 0). Left: the height of the tip, \(\cos\theta\), for both. Middle: their torques. Right: a pendulum with \(m=1.2\) and a torque disturbance of 0.1 times a standard normal value at each step (NumPy seed 1 for each controller): the MPPI torques replayed open loop, the MPPI trajectory tracked by time-varying LQR (\(Q=\operatorname{diag}(1,0.1)\), \(R=0.01\), linearized along it), and MPC with MPPI on the nominal model.

16. Deep Q-learning {#16-deep-q-learning}

Script: ch16_dqn.py, the only script that uses Gymnasium.

Local figure Data and construction

rl-dqn-overestimation.png No randomness. Action values fitted by least squares with polynomials of degree 6 to the true values at the integer states from \(-6\) to 6, each fit leaving out a different pair of adjacent interior points, and evaluated at 601 states from \(-6\) to 6. Left: ten fits to \(\sin s\) (gray), \(\sin s\), the maximum of the fits (red), and the double estimate (dashed), the value under a second set of fits, with the omitted pairs shifted by five, of the action that the first set maximizes. Right: the average error over the 601 states of the maximum and of the double estimate, for 1 to 10 actions and true values \(\sin s\) (solid) and \(2e^{-s^2}\) (dashed).

rl-dqn-cartpole.png Gymnasium’s CartPole-v1. DQN with an MLP from the 4 state variables through two hidden layers of 64 ReLU units to 2 action values, Adam at learning rate \(10^{-3}\), the Huber loss (smooth_l1_loss) on targets \(r+0.99\,(1-\text{terminated})\max_a\hat q_{\text{target}}(s',a)\), which bootstrap through truncation at 500 steps, a replay memory of 50,000, minibatches of 64 drawn uniformly with replacement, ε decreasing linearly from 1 to 0.05 over the first 10,000 steps, and one update per step from step 1,000, for 30,000 steps. Two runs, with the target network copied every 500 and every 100 steps, each from torch seed 0 and NumPy seed 0, with the environment reset with seed 0 at the start and without a seed afterward. Every 1,000 steps, 10 greedy episodes on a second environment reset with seeds 1000–1009. Left: their mean length (at most 500). Right: the mean of \(\max_a\hat q(s_0,a)\) at their start states (solid) and their mean discounted return with \(\gamma=0.99\) (dashed), with the largest possible discounted return, \((1-0.99^{500})/(1-0.99)\), dotted.

Local figure	Data and construction
rl-dqn-overestimation.png	No randomness. Action values fitted by least squares with polynomials of degree 6 to the true values at the integer states from \(-6\) to 6, each fit leaving out a different pair of adjacent interior points, and evaluated at 601 states from \(-6\) to 6. Left: ten fits to \(\sin s\) (gray), \(\sin s\), the maximum of the fits (red), and the double estimate (dashed), the value under a second set of fits, with the omitted pairs shifted by five, of the action that the first set maximizes. Right: the average error over the 601 states of the maximum and of the double estimate, for 1 to 10 actions and true values \(\sin s\) (solid) and \(2e^{-s^2}\) (dashed).
rl-dqn-cartpole.png	Gymnasium’s `CartPole-v1`. DQN with an MLP from the 4 state variables through two hidden layers of 64 ReLU units to 2 action values, Adam at learning rate \(10^{-3}\), the Huber loss (`smooth_l1_loss`) on targets \(r+0.99\,(1-\text{terminated})\max_a\hat q_{\text{target}}(s',a)\), which bootstrap through truncation at 500 steps, a replay memory of 50,000, minibatches of 64 drawn uniformly with replacement, ε decreasing linearly from 1 to 0.05 over the first 10,000 steps, and one update per step from step 1,000, for 30,000 steps. Two runs, with the target network copied every 500 and every 100 steps, each from torch seed 0 and NumPy seed 0, with the environment reset with seed 0 at the start and without a seed afterward. Every 1,000 steps, 10 greedy episodes on a second environment reset with seeds 1000–1009. Left: their mean length (at most 500). Right: the mean of \(\max_a\hat q(s_0,a)\) at their start states (solid) and their mean discounted return with \(\gamma=0.99\) (dashed), with the largest possible discounted return, \((1-0.99^{500})/(1-0.99)\), dotted.

17. Distributional reinforcement learning {#17-distributional-reinforcement-learning}

Script: ch17_distributional.py.

Local figure Data and construction

rl-dist-projections.png No randomness. Left: a target distribution with atoms \(-2.7\), \(-0.4\), 1.3, and 2.35 and probabilities 0.2, 0.35, 0.3, and 0.15, projected onto nine atoms from \(-4\) to 4 by splitting each atom’s mass between its two neighbors in proportion to proximity. Right: the cumulative distribution of \(0.5\,\mathcal N(-1.8,0.7^2)+0.5\,\mathcal N(1.5,1)\) (SciPy’s stats.norm) on 1,000 points from \(-5\) to 5, and the step function of 8 equally weighted atoms at its quantiles \(\tau=1/16\), 3/16, …, 15/16, located on the grid.

rl-dist-bimodal.png The chapter’s process: from the start, reward 0 and one of two chains of five states with probability 1/2 each; every step along a chain pays \(\pm1\) with probability 1/2, plus \(+10\) on leaving the last state of chain A and \(-10\) on leaving the last state of chain B; \(\gamma=0.9\). The exact return distribution comes from enumerating the 64 outcomes. Categorical TD on 51 atoms from \(-15\) to 15, each state’s distribution starting as a point mass at 0 and moving toward the projected target with step size \(n^{-0.7}\) at the \(n\)th visit; quantile TD with 32 quantiles at \(\tau=(2i-1)/64\), starting at 0, each moved by \(\max(0.02,2/\sqrt n)\) times \(\tau\) minus the fraction of the 32 target atoms \(r+\gamma\theta_j(s')\) below it; 20,000 episodes (NumPy seed 0). Left: the exact distribution projected onto the 51 atoms and categorical TD’s distribution at the start, with the mean, 0, dotted. Right: the exact, categorical, and quantile cumulative distributions on 3,001 points from \(-15\) to 15.

Local figure	Data and construction
rl-dist-projections.png	No randomness. Left: a target distribution with atoms \(-2.7\), \(-0.4\), 1.3, and 2.35 and probabilities 0.2, 0.35, 0.3, and 0.15, projected onto nine atoms from \(-4\) to 4 by splitting each atom’s mass between its two neighbors in proportion to proximity. Right: the cumulative distribution of \(0.5\,\mathcal N(-1.8,0.7^2)+0.5\,\mathcal N(1.5,1)\) (SciPy’s `stats.norm`) on 1,000 points from \(-5\) to 5, and the step function of 8 equally weighted atoms at its quantiles \(\tau=1/16\), 3/16, …, 15/16, located on the grid.
rl-dist-bimodal.png	The chapter’s process: from the start, reward 0 and one of two chains of five states with probability 1/2 each; every step along a chain pays \(\pm1\) with probability 1/2, plus \(+10\) on leaving the last state of chain A and \(-10\) on leaving the last state of chain B; \(\gamma=0.9\). The exact return distribution comes from enumerating the 64 outcomes. Categorical TD on 51 atoms from \(-15\) to 15, each state’s distribution starting as a point mass at 0 and moving toward the projected target with step size \(n^{-0.7}\) at the \(n\)th visit; quantile TD with 32 quantiles at \(\tau=(2i-1)/64\), starting at 0, each moved by \(\max(0.02,2/\sqrt n)\) times \(\tau\) minus the fraction of the 32 target atoms \(r+\gamma\theta_j(s')\) below it; 20,000 episodes (NumPy seed 0). Left: the exact distribution projected onto the 51 atoms and categorical TD’s distribution at the start, with the mean, 0, dotted. Right: the exact, categorical, and quantile cumulative distributions on 3,001 points from \(-15\) to 15.

18. Data-efficient and scalable value-based agents {#18-data-efficient-and-scalable-value-based-agents}

Script: ch18_rainbow.py.

Local figure	Data and construction
rl-rainbow-replay.png	All draws come from one generator (NumPy seed 0). Left: the blind cliffwalk with \(n=4\), 6, …, 14 states, whose rewarded (“right”) action in each state is drawn at random; the wrong action ends the episode with reward 0, and the right action in the last state ends it with reward 1; \(\gamma=1-1/n\). The memory holds the transitions of \(2^n\) episodes of uniformly random actions, with one successful episode appended if none succeeded. Q-learning from 0 with step size 0.25 replays transitions drawn uniformly or in proportion to their priorities (a sum tree, every priority 1 at first and \(\lvert\delta\rvert+10^{-4}\) after each replay) until the mean squared error from the true values falls below 1% of its initial value, with at most 3 million replays. Medians over 5 memories drawn afresh for each \(n\) and method, with the median memory size dotted, on a logarithmic axis. Right: the cliff walking of chapter 7 with ε-greedy behavior (\(\varepsilon=0.1\), ties broken at random), step size 0.1, and uncorrected, undiscounted \(n\)-step Q-learning targets (the next \(n\) rewards plus \(\max_a Q\) at the state reached), for \(n=1\) and 10 with 500 episodes each; the greedy paths afterward, at most 60 steps. The script prints the memory sizes and the medians.

rl-rainbow-replay.png

All draws come from one generator (NumPy seed 0). Left: the blind cliffwalk with \(n=4\), 6, …, 14 states, whose rewarded (“right”) action in each state is drawn at random; the wrong action ends the episode with reward 0, and the right action in the last state ends it with reward 1; \(\gamma=1-1/n\). The memory holds the transitions of \(2^n\) episodes of uniformly random actions, with one successful episode appended if none succeeded. Q-learning from 0 with step size 0.25 replays transitions drawn uniformly or in proportion to their priorities (a sum tree, every priority 1 at first and \(\lvert\delta\rvert+10^{-4}\) after each replay) until the mean squared error from the true values falls below 1% of its initial value, with at most 3 million replays. Medians over 5 memories drawn afresh for each \(n\) and method, with the median memory size dotted, on a logarithmic axis. Right: the cliff walking of chapter 7 with ε-greedy behavior (\(\varepsilon=0.1\), ties broken at random), step size 0.1, and uncorrected, undiscounted \(n\)-step Q-learning targets (the next \(n\) rewards plus \(\max_a Q\) at the state reached), for \(n=1\) and 10 with 500 episodes each; the greedy paths afterward, at most 60 steps. The script prints the memory sizes and the medians.

19. Deep actor-critic and distributed RL {#19-deep-actor-critic-and-distributed-rl}

Script: ch19_actor_critic.py.

Local figure	Data and construction
rl-ac-lag.png	Left: a batched cart-pole in NumPy with the classic equations of chapter 15 (force \(\pm10\), steps of 0.02 s), termination when the cart passes \(\pm2.4\) or the pole \(\pm12°\), truncation at 500 steps, and starts uniform on \([-0.05,0.05]^4\) (NumPy seed 0). Four configurations, actors that use the current parameters and actors that use those of 32 updates earlier with no correction, with V-trace, or with PPO’s clipped objective, each with 6 agents, trained together as a batch of 24 (generator seed 0 for initialization and action sampling). Each agent has an actor and a critic, MLPs with two hidden layers of 64 tanh units, uniform initialization with bound \(1/\sqrt{\text{fan-in}}\) (0.01 times that for the last layer) and zero biases, inputs divided by (2.4, 2, 0.21, 2), and the critic’s output multiplied by 10. Each update uses 16 environments and 16-step rollouts, reward 1 per step, and \(\gamma=0.99\). The critic’s targets are its values plus a backward sum of \(\rho\,\delta\) terms with trace coefficients \(0.95\rho\), where \(\rho=\min(1,\pi/\mu)\) with V-trace and 1 otherwise; the actor’s advantage is that sum or, with V-trace, \(\rho\) times the one-step error toward the next target; the clipped configuration clips the ratio of the current to the acting policy to \([0.8,1.2]\). Entropy bonus 0.01, value loss weight 0.5, and Adam, written out in the script, at learning rate \(10^{-3}\), for 1,500 updates. Every 25 updates, the mean length of each agent’s last 10 finished episodes, averaged over the 6 agents of a configuration. Right: a random MDP with 6 states, 3 actions, flat-Dirichlet transition rows, standard normal rewards, a behavior \(\mu\) with flat-Dirichlet rows, a target that is the softmax of 1.5 times standard normal logits, and \(\gamma=0.9\) (NumPy seed 1); for 81 truncation levels \(\bar\rho\) from \(10^{-2}\) to \(10^2\), the value of \(\pi_{\bar\rho}\propto\min(\bar\rho\,\mu,\pi)\), normalized as \((v-v_\mu)/(v_\pi-v_\mu)\), one line per state. The script prints, for each configuration, the average over the last 8 logged points of each agent.

rl-ac-lag.png

Left: a batched cart-pole in NumPy with the classic equations of chapter 15 (force \(\pm10\), steps of 0.02 s), termination when the cart passes \(\pm2.4\) or the pole \(\pm12°\), truncation at 500 steps, and starts uniform on \([-0.05,0.05]^4\) (NumPy seed 0). Four configurations, actors that use the current parameters and actors that use those of 32 updates earlier with no correction, with V-trace, or with PPO’s clipped objective, each with 6 agents, trained together as a batch of 24 (generator seed 0 for initialization and action sampling). Each agent has an actor and a critic, MLPs with two hidden layers of 64 tanh units, uniform initialization with bound \(1/\sqrt{\text{fan-in}}\) (0.01 times that for the last layer) and zero biases, inputs divided by (2.4, 2, 0.21, 2), and the critic’s output multiplied by 10. Each update uses 16 environments and 16-step rollouts, reward 1 per step, and \(\gamma=0.99\). The critic’s targets are its values plus a backward sum of \(\rho\,\delta\) terms with trace coefficients \(0.95\rho\), where \(\rho=\min(1,\pi/\mu)\) with V-trace and 1 otherwise; the actor’s advantage is that sum or, with V-trace, \(\rho\) times the one-step error toward the next target; the clipped configuration clips the ratio of the current to the acting policy to \([0.8,1.2]\). Entropy bonus 0.01, value loss weight 0.5, and Adam, written out in the script, at learning rate \(10^{-3}\), for 1,500 updates. Every 25 updates, the mean length of each agent’s last 10 finished episodes, averaged over the 6 agents of a configuration. Right: a random MDP with 6 states, 3 actions, flat-Dirichlet transition rows, standard normal rewards, a behavior \(\mu\) with flat-Dirichlet rows, a target that is the softmax of 1.5 times standard normal logits, and \(\gamma=0.9\) (NumPy seed 1); for 81 truncation levels \(\bar\rho\) from \(10^{-2}\) to \(10^2\), the value of \(\pi_{\bar\rho}\propto\min(\bar\rho\,\mu,\pi)\), normalized as \((v-v_\mu)/(v_\pi-v_\mu)\), one line per state. The script prints, for each configuration, the average over the last 8 logged points of each agent.

20. Trust regions and proximal policy optimization {#20-trust-regions-and-proximal-policy-optimization}

Script: ch20_ppo.py.

Local figure	Data and construction
rl-ppo-clip.png	Left and middle: no data are used; the objective \(\min(rA,\operatorname{clip}(r,0.8,1.2)A)\) for \(A=1\) and \(A=-1\) at 401 ratios from 0 to 2, with the unclipped \(rA\) dotted and the region without gradient shaded. Right: a random MDP with 20 states, 4 actions, transition rows from a Dirichlet distribution with parameters 0.1, rewards equal to the cubes of uniform numbers on \([0,1]\) (NumPy seed 0), \(\gamma=0.9\), and a uniform initial distribution, with \(J(\pi^)\) from 2,000 sweeps of value iteration. A tabular softmax policy starts with preference 8 on each state’s worst action under the uniform policy (the smallest \(q\)) and 0 on the others; exact policy gradient with \(\eta=1\) and 10, natural policy gradient (\(\theta\leftarrow\theta+\eta A/(1-\gamma)\)) with \(\eta=0.1\) and 1, and policy iteration, each for 1,000 iterations. The gap \(J(\pi^)-J(\pi_k)\), floored at \(10^{-16}\), on log–log axes cut at \(10^{-9}\) and 30.

rl-ppo-clip.png

Left and middle: no data are used; the objective \(\min(rA,\operatorname{clip}(r,0.8,1.2)A)\) for \(A=1\) and \(A=-1\) at 401 ratios from 0 to 2, with the unclipped \(rA\) dotted and the region without gradient shaded. Right: a random MDP with 20 states, 4 actions, transition rows from a Dirichlet distribution with parameters 0.1, rewards equal to the cubes of uniform numbers on \([0,1]\) (NumPy seed 0), \(\gamma=0.9\), and a uniform initial distribution, with \(J(\pi^*)\) from 2,000 sweeps of value iteration. A tabular softmax policy starts with preference 8 on each state’s worst action under the uniform policy (the smallest \(q\)) and 0 on the others; exact policy gradient with \(\eta=1\) and 10, natural policy gradient (\(\theta\leftarrow\theta+\eta A/(1-\gamma)\)) with \(\eta=0.1\) and 1, and policy iteration, each for 1,000 iterations. The gap \(J(\pi^*)-J(\pi_k)\), floored at \(10^{-16}\), on log–log axes cut at \(10^{-9}\) and 30.

21. Continuous control and maximum-entropy RL {#21-continuous-control-and-maximum-entropy-rl}

Script: ch21_maxent.py.

Local figure	Data and construction
rl-sac-maxent.png	Left (no randomness): the cliff-walking grid of chapter 7 with \(\gamma=0.99\), reward \(-1\) per step, and \(-100\) for stepping into the cliff, which here ends the episode, as reaching the goal does. Soft value iteration, \(V(s)=\alpha\log\sum_a\exp(Q(s,a)/\alpha)\), for 3,000 iterations at temperatures \(\alpha=0.01\), 0.5, and 0.65; the soft-optimal policy \(\pi(a\mid s)=\exp((Q(s,a)-V(s))/\alpha)\); and its expected visits per episode from the start, by a linear solve, drawn as \(\log_{10}\) of the visits clipped at \(10^{-3}\) (Blues colormap from \(-3\) to 1) with the cliff shaded red. Right: a true action value \(-(a-0.3)^2\) at 401 actions in \([-1,1]\) and two critics with errors \(\sigma u\) and \(\sigma(\rho u+\sqrt{1-\rho^2}\,v)\), where \(u\) and \(v\) are independent smooth random functions, sums of 8 cosines with frequencies uniform on \([0,6]\), phases uniform on \([0,2\pi]\), and standard normal weights, divided by 2 (NumPy seed 0). For 13 values of \(\sigma\) from 0.02 to 0.5 and \(\rho=0\) (solid) and 0.8 (dashed), with 5,000 draws each, the mean error at the action that maximizes critic 1 of critic 1 (DDPG), of critic 2 (double), and of the smaller of the two (TD3); critic 1 is drawn only for \(\rho=0\).

rl-sac-maxent.png

Left (no randomness): the cliff-walking grid of chapter 7 with \(\gamma=0.99\), reward \(-1\) per step, and \(-100\) for stepping into the cliff, which here ends the episode, as reaching the goal does. Soft value iteration, \(V(s)=\alpha\log\sum_a\exp(Q(s,a)/\alpha)\), for 3,000 iterations at temperatures \(\alpha=0.01\), 0.5, and 0.65; the soft-optimal policy \(\pi(a\mid s)=\exp((Q(s,a)-V(s))/\alpha)\); and its expected visits per episode from the start, by a linear solve, drawn as \(\log_{10}\) of the visits clipped at \(10^{-3}\) (Blues colormap from \(-3\) to 1) with the cliff shaded red. Right: a true action value \(-(a-0.3)^2\) at 401 actions in \([-1,1]\) and two critics with errors \(\sigma u\) and \(\sigma(\rho u+\sqrt{1-\rho^2}\,v)\), where \(u\) and \(v\) are independent smooth random functions, sums of 8 cosines with frequencies uniform on \([0,6]\), phases uniform on \([0,2\pi]\), and standard normal weights, divided by 2 (NumPy seed 0). For 13 values of \(\sigma\) from 0.02 to 0.5 and \(\rho=0\) (solid) and 0.8 (dashed), with 5,000 draws each, the mean error at the action that maximizes critic 1 of critic 1 (DDPG), of critic 2 (double), and of the smaller of the two (TD3); critic 1 is drawn only for \(\rho=0\).

22. Exploration in deep RL {#22-exploration-in-deep-rl}

Script: ch22_exploration.py.

Local figure	Data and construction
rl-explore.png	Left: Deep Sea of size \(N\), an \(N\times N\) grid in which the agent starts at the top left and moves down one row per step for \(N\) steps; in each cell a randomly chosen one of the two actions moves right at a cost of \(0.01/N\) and the other moves left, staying in the column at the edges, and moving right from the bottom-right cell pays 1. Agents: ε-greedy Q-learning (\(\varepsilon=0.1\), step size 0.1); a planner that uses backward induction with the known transitions and the mean observed rewards plus \(1/\sqrt{n+1}\) (optimism); and one that adds \(z/\sqrt{n+1}\), \(z\) standard normal, instead (posterior sampling); ties are broken at random. Episodes until the reward is first found, at most 8,000; medians of 9 runs for ε-greedy (\(N=3,\dots,8\)) and of 3 runs for the others (\(N=4\), 8, 16, 24, 32, 48, and 64), each run with a new environment, from a generator for each agent and size (NumPy seed \(100+N\)). Sizes whose median exceeds the limit are marked × at 8,000, and the dotted line is \(2^N\). Middle and right: the state space \([0,1]^2\) with experience from three sources: clusters at (0.25, 0.25) with probability 0.7 and at (0.25, 0.75) with probability 0.03, both with standard deviation 0.04 and next states moved by actions from \(\mathcal N(0,0.02^2)\), and a “noisy TV”, uniform on \([0.7,0.9]\times[0.1,0.3]\), with probability 0.27, whose next state is a new uniform point (NumPy seed 0). Random network distillation, with a fixed random target and a trained predictor, both MLPs from 2 inputs through two hidden layers of 64 ReLU units to 16 outputs, and a forward model of the same shape from state and action to the change of state, trained together with Adam at learning rate \(10^{-3}\) for 10,000 steps of 128 fresh samples (torch seed 0). The maps show \(\log_{10}\) of the squared error of each on a \(101\times101\) grid (magma colormap); the forward model’s error uses actions from torch’s generator and, in the TV, the average over 50 uniform next frames. The script prints the medians of the left panel.

rl-explore.png

Left: Deep Sea of size \(N\), an \(N\times N\) grid in which the agent starts at the top left and moves down one row per step for \(N\) steps; in each cell a randomly chosen one of the two actions moves right at a cost of \(0.01/N\) and the other moves left, staying in the column at the edges, and moving right from the bottom-right cell pays 1. Agents: ε-greedy Q-learning (\(\varepsilon=0.1\), step size 0.1); a planner that uses backward induction with the known transitions and the mean observed rewards plus \(1/\sqrt{n+1}\) (optimism); and one that adds \(z/\sqrt{n+1}\), \(z\) standard normal, instead (posterior sampling); ties are broken at random. Episodes until the reward is first found, at most 8,000; medians of 9 runs for ε-greedy (\(N=3,\dots,8\)) and of 3 runs for the others (\(N=4\), 8, 16, 24, 32, 48, and 64), each run with a new environment, from a generator for each agent and size (NumPy seed \(100+N\)). Sizes whose median exceeds the limit are marked × at 8,000, and the dotted line is \(2^N\). Middle and right: the state space \([0,1]^2\) with experience from three sources: clusters at (0.25, 0.25) with probability 0.7 and at (0.25, 0.75) with probability 0.03, both with standard deviation 0.04 and next states moved by actions from \(\mathcal N(0,0.02^2)\), and a “noisy TV”, uniform on \([0.7,0.9]\times[0.1,0.3]\), with probability 0.27, whose next state is a new uniform point (NumPy seed 0). Random network distillation, with a fixed random target and a trained predictor, both MLPs from 2 inputs through two hidden layers of 64 ReLU units to 16 outputs, and a forward model of the same shape from state and action to the change of state, trained together with Adam at learning rate \(10^{-3}\) for 10,000 steps of 128 fresh samples (torch seed 0). The maps show \(\log_{10}\) of the squared error of each on a \(101\times101\) grid (magma colormap); the forward model’s error uses actions from torch’s generator and, in the TV, the average over 50 uniform next frames. The script prints the medians of the left panel.

23. Model-based RL and world models {#23-model-based-rl-and-world-models}

Script: ch23_models.py.

Local figure	Data and construction
rl-model-errors.png	A pendulum with \(\theta=0\) upright, \(\dot\theta\leftarrow\dot\theta+(15\sin\theta+3u)\,\Delta t\) clipped to \([-8,8]\) and \(\theta\leftarrow\theta+\dot\theta\,\Delta t\) with \(\Delta t=0.05\), observed as \((\cos\theta,\sin\theta,\dot\theta/8)\). Training data: 2,000 transitions from angles whose magnitude is uniform on \([2,\pi]\), with a random sign, angular velocities uniform on \([-2,2]\), and torques uniform on \([-2,2]\) (NumPy seed 0). An ensemble of five MLPs from the observation and \(u/2\) through two hidden layers of 64 tanh units to the change of the observation, each trained on its own bootstrap resample, drawn with NumPy, for 1,500 Adam steps at learning rate \(3\times10^{-3}\) on the squared error of minibatches of 256 drawn from the resample with replacement (torch seed 0). 500 open-loop rollouts of 30 steps with uniform random torques, from starts like the training data and from angles uniform on \([-0.5,0.5]\) near the top; the error is the distance between the ensemble’s mean prediction and the true observation, and the disagreement is the square root of the summed variance across the ensemble. Left: the mean error (solid) and disagreement (dashed) against the number of steps, on a logarithmic axis. Right: the disagreement and the error after 10 steps of each rollout, on log–log axes with the diagonal dotted.

rl-model-errors.png

A pendulum with \(\theta=0\) upright, \(\dot\theta\leftarrow\dot\theta+(15\sin\theta+3u)\,\Delta t\) clipped to \([-8,8]\) and \(\theta\leftarrow\theta+\dot\theta\,\Delta t\) with \(\Delta t=0.05\), observed as \((\cos\theta,\sin\theta,\dot\theta/8)\). Training data: 2,000 transitions from angles whose magnitude is uniform on \([2,\pi]\), with a random sign, angular velocities uniform on \([-2,2]\), and torques uniform on \([-2,2]\) (NumPy seed 0). An ensemble of five MLPs from the observation and \(u/2\) through two hidden layers of 64 tanh units to the change of the observation, each trained on its own bootstrap resample, drawn with NumPy, for 1,500 Adam steps at learning rate \(3\times10^{-3}\) on the squared error of minibatches of 256 drawn from the resample with replacement (torch seed 0). 500 open-loop rollouts of 30 steps with uniform random torques, from starts like the training data and from angles uniform on \([-0.5,0.5]\) near the top; the error is the distance between the ensemble’s mean prediction and the true observation, and the disagreement is the square root of the summed variance across the ensemble. Left: the mean error (solid) and disagreement (dashed) against the number of steps, on a logarithmic axis. Right: the disagreement and the error after 10 steps of each rollout, on log–log axes with the diagonal dotted.

24. Planning with learned models {#24-planning-with-learned-models}

Script: ch24_search.py.

Local figure Data and construction

rl-search.png Left: tic-tac-toe; the nonterminal positions reachable from the empty board in which the moves do not all have the same exact value, of which 400 are drawn without replacement (NumPy seed 0). A simulated network gives each position a prior, the softmax of twice the exact values of the moves plus \(\sigma\) times standard normal noise, and a value, the exact value plus \(\sigma\) times standard normal noise, clipped to \([-1,1]\); the noise comes from a generator seeded by Python’s hash of the position and 0, so it is fixed for each position. PUCT with \(c=1.5\), exact values at terminal positions, and backups with alternating signs plays the most visited move. The fraction of test positions in which that move is optimal, for 1, 2, 4, …, 256 simulations and \(\sigma=1\), 2, and 4, with the fraction for the prior’s most likely move dotted. Right: one-step problems with 16 actions, values \(q\sim\mathcal N(0,1)\), and logits \(0.5q+\sqrt{0.75}\,z\), 5,000 for each budget of 1, 2, 4, 8, 16, and 32 simulations, all from one generator (NumPy seed 1). PUCT with \(c=1.25\), whose simulations each return the exact value, plays the most visited action; Gumbel search takes the \(\min(n,16)\) actions with the largest Gumbel noise plus logit and plays the one among them with the largest Gumbel noise plus logit plus \(50q\). The average value of the chosen action, with the prior policy’s expected value (dashed) and the best action’s value (dotted). The script prints the fractions of the left panel.

Local figure	Data and construction
rl-search.png	Left: tic-tac-toe; the nonterminal positions reachable from the empty board in which the moves do not all have the same exact value, of which 400 are drawn without replacement (NumPy seed 0). A simulated network gives each position a prior, the softmax of twice the exact values of the moves plus \(\sigma\) times standard normal noise, and a value, the exact value plus \(\sigma\) times standard normal noise, clipped to \([-1,1]\); the noise comes from a generator seeded by Python’s `hash` of the position and 0, so it is fixed for each position. PUCT with \(c=1.5\), exact values at terminal positions, and backups with alternating signs plays the most visited move. The fraction of test positions in which that move is optimal, for 1, 2, 4, …, 256 simulations and \(\sigma=1\), 2, and 4, with the fraction for the prior’s most likely move dotted. Right: one-step problems with 16 actions, values \(q\sim\mathcal N(0,1)\), and logits \(0.5q+\sqrt{0.75}\,z\), 5,000 for each budget of 1, 2, 4, 8, 16, and 32 simulations, all from one generator (NumPy seed 1). PUCT with \(c=1.25\), whose simulations each return the exact value, plays the most visited action; Gumbel search takes the \(\min(n,16)\) actions with the largest Gumbel noise plus logit and plays the one among them with the largest Gumbel noise plus logit plus \(50q\). The average value of the chosen action, with the prior policy’s expected value (dashed) and the best action’s value (dotted). The script prints the fractions of the left panel.

25. Imitation learning and inverse RL {#25-imitation-learning-and-inverse-rl}

Script: ch25_imitation.py.

Local figure	Data and construction
rl-imitation.png	Left: a car moving at speed 0.5 with time steps of 0.05, whose heading changes by \(u\,\Delta t\) plus a gust of 0.05 times a standard normal value, on a circular track of radius 1; it leaves the track when its distance from the circle exceeds 0.25, and a lap attempt lasts at most 300 steps. The expert steers toward the point 0.4 rad ahead on the circle with gain 4, clipped to \([-3,3]\). Policies are 5-nearest-neighbor regressions of the expert’s steering on the radius and the sine and cosine of the heading relative to the track’s tangent. Behavior cloning: 1,000 labels from expert laps without gusts that start exactly on the track with the tangent heading. DAgger: 333 labeled expert steps under gusts, then laps of the current learner under gusts, labeled by the expert, until there are 1,000 labels (NumPy seed 0). Laps under gusts start at a random angle with radius \(1\pm0.1\) and heading within 0.3 rad of the tangent. Three laps of each policy under gusts (NumPy seeds 100, 101, and 102 for both), with × where the car left the track, over the band of radii from 0.75 to 1.25. Right: maximum-entropy inverse RL on a \(10\times10\) map (NumPy seed 0, a new generator) of road, grass, mud, and water with probabilities 0.3, 0.35, 0.2, and 0.15 and one goal cell, with rewards per step \(-0.1\), \(-0.3\), \(-1\), \(-3\), and \(+1\); five actions (stay and four moves) and horizon 30. Demonstrations follow the finite-horizon soft-optimal policy from uniformly random cells; the mean terrain counts of 20 and of 1,000 demonstrations are matched by 300 gradient steps of size 0.05 on the weights, with the expected counts of the soft-optimal policy for the current weights computed exactly from a uniform start. The recovered weights are shifted so that road has its true value.

rl-imitation.png

Left: a car moving at speed 0.5 with time steps of 0.05, whose heading changes by \(u\,\Delta t\) plus a gust of 0.05 times a standard normal value, on a circular track of radius 1; it leaves the track when its distance from the circle exceeds 0.25, and a lap attempt lasts at most 300 steps. The expert steers toward the point 0.4 rad ahead on the circle with gain 4, clipped to \([-3,3]\). Policies are 5-nearest-neighbor regressions of the expert’s steering on the radius and the sine and cosine of the heading relative to the track’s tangent. Behavior cloning: 1,000 labels from expert laps without gusts that start exactly on the track with the tangent heading. DAgger: 333 labeled expert steps under gusts, then laps of the current learner under gusts, labeled by the expert, until there are 1,000 labels (NumPy seed 0). Laps under gusts start at a random angle with radius \(1\pm0.1\) and heading within 0.3 rad of the tangent. Three laps of each policy under gusts (NumPy seeds 100, 101, and 102 for both), with × where the car left the track, over the band of radii from 0.75 to 1.25. Right: maximum-entropy inverse RL on a \(10\times10\) map (NumPy seed 0, a new generator) of road, grass, mud, and water with probabilities 0.3, 0.35, 0.2, and 0.15 and one goal cell, with rewards per step \(-0.1\), \(-0.3\), \(-1\), \(-3\), and \(+1\); five actions (stay and four moves) and horizon 30. Demonstrations follow the finite-horizon soft-optimal policy from uniformly random cells; the mean terrain counts of 20 and of 1,000 demonstrations are matched by 300 gradient steps of size 0.05 on the weights, with the expected counts of the soft-optimal policy for the current weights computed exactly from a uniform start. The recovered weights are shifted so that road has its true value.

26. Offline reinforcement learning {#26-offline-reinforcement-learning}

Script: ch26_offline.py.

Local figure	Data and construction
rl-offline.png	2,000 logged one-step samples (NumPy seed 0): contexts \(s\) uniform on \([-1,1]\), behavior actions \(a=0.5s+0.1z\), and rewards equal to the true value \(1-(a-s)^2-5\max(0,\lvert a-s/2\rvert-0.3)^2\) plus \(0.1z'\), with \(z\) and \(z'\) standard normal. Two MLPs from \((s,a)\) through two hidden layers of 64 ReLU units to a value, each from torch seed 1, trained by Adam at learning rate \(10^{-3}\) for 2,000 steps of 256 samples drawn with replacement on the squared error to the rewards; one also minimizes the CQL penalty, 0.5 times the log-sum-exp of its values at 41 actions from \(-2\) to 2 minus its value at the logged action. Panels: contexts 0.8 and \(-0.4\), with the true value and both fits at 401 actions from \(-2\) to 2, each fit’s maximizing action marked, and the behavior’s central range \(0.5s\pm0.2\) shaded.

rl-offline.png

2,000 logged one-step samples (NumPy seed 0): contexts \(s\) uniform on \([-1,1]\), behavior actions \(a=0.5s+0.1z\), and rewards equal to the true value \(1-(a-s)^2-5\max(0,\lvert a-s/2\rvert-0.3)^2\) plus \(0.1z'\), with \(z\) and \(z'\) standard normal. Two MLPs from \((s,a)\) through two hidden layers of 64 ReLU units to a value, each from torch seed 1, trained by Adam at learning rate \(10^{-3}\) for 2,000 steps of 256 samples drawn with replacement on the squared error to the rewards; one also minimizes the CQL penalty, 0.5 times the log-sum-exp of its values at 41 actions from \(-2\) to 2 minus its value at the logged action. Panels: contexts 0.8 and \(-0.4\), with the true value and both fits at 401 actions from \(-2\) to 2, each fit’s maximizing action marked, and the behavior’s central range \(0.5s\pm0.2\) shaded.

27. Multi-agent RL and self-play {#27-multi-agent-rl-and-self-play}

Script: ch27_games.py.

Local figure	Data and construction
rl-games.png	No randomness. The biased rock–paper–scissors of the chapter’s first code, whose row player receives \((0,-1,2)\) with rock, \((1,0,-1)\) with paper, and \((-2,1,0)\) with scissors against rock, paper, and scissors; its equilibrium is \((0.25,0.5,0.25)\). The row player starts at \((0.6,0.2,0.2)\) and the column player at \((0.2,0.2,0.6)\). Simultaneous entropic mirror descent with step size 0.1, and the regularized dynamics that also pull each strategy toward a magnet with weight 0.2, a fixed uniform magnet or one reset to the current strategies every 200 steps, for 10,000 steps. Left: the row player’s strategy over the first 3,000 steps in the simplex, with the start, the equilibrium (star), and the final point of the fixed-magnet dynamics (square). Right: the exploitability \(\max_i(Ay)_i-\min_j(x^\top A)_j\) of the current strategies of each method and of mirror descent’s running averages, on log–log axes cut at \(10^{-6}\) and 5. The script prints the final exploitabilities and the fixed-magnet point.

rl-games.png

No randomness. The biased rock–paper–scissors of the chapter’s first code, whose row player receives \((0,-1,2)\) with rock, \((1,0,-1)\) with paper, and \((-2,1,0)\) with scissors against rock, paper, and scissors; its equilibrium is \((0.25,0.5,0.25)\). The row player starts at \((0.6,0.2,0.2)\) and the column player at \((0.2,0.2,0.6)\). Simultaneous entropic mirror descent with step size 0.1, and the regularized dynamics that also pull each strategy toward a magnet with weight 0.2, a fixed uniform magnet or one reset to the current strategies every 200 steps, for 10,000 steps. Left: the row player’s strategy over the first 3,000 steps in the simplex, with the start, the equilibrium (star), and the final point of the fixed-magnet dynamics (square). Right: the exploitability \(\max_i(Ay)_i-\min_j(x^\top A)_j\) of the current strategies of each method and of mirror descent’s running averages, on log–log axes cut at \(10^{-6}\) and 5. The script prints the final exploitabilities and the fixed-magnet point.

28. Reinforcement learning for language models and reasoning {#28-reinforcement-learning-for-language-models-and-reasoning}

Script: ch28_llm.py.

Local figure	Data and construction
rl-llm.png	Left: the toy of the chapter’s first code (NumPy seed 0): 10 responses, a reference policy that is the softmax of standard normal logits, rewards uniform on \([0,1]\), \(\beta=0.5\), and the KL-regularized optimum \(\pi^\propto\pi_{\text{ref}}\,e^{r/\beta}\); the script draws and discards \(100{,}000\times16\) samples from \(\pi^\) to keep its random stream aligned with the chapter’s code. A tabular softmax policy starts at the reference and takes 4,000 steps with learning rate 0.5 on groups of 256 sampled responses with the group mean as baseline, with the penalty \(\beta\log(\pi/\pi_{\text{ref}})\) subtracted from the reward (\(k_1\) in the reward), or with \(\beta\) times the sampled gradient of \(k_1=\log(\pi/\pi_{\text{ref}})\) or of \(k_3=\pi_{\text{ref}}/\pi-1-\log(\pi_{\text{ref}}/\pi)\) subtracted as a loss. Bars: the reference, \(\pi^\), and the two loss versions, over the responses sorted by reward (the tick labels); instead of drawing the reward version, the script prints its total variation from \(\pi^\). Right: 4 answers, the first correct (reward 1), each followed by up to 40 tokens, each of which ends the response with probability given by the sigmoid of a logit for that answer and position, starting at \(-2\); the answer logits start at 0. Groups of 8 responses; GRPO standardizes the rewards within the group and weights each response by \(1/(8\times\text{its length})\), and Dr. GRPO only centers them and weights each response by \(1/(8\times41)\); learning rates 2 (GRPO) and 20 (Dr. GRPO), a tenth of that for the answer logits, and 1,000 steps. Every 10 steps, the expected length of the correct answer’s responses and the mean over the incorrect ones, averaged over 20 runs (NumPy seeds 0–19).

rl-llm.png

Left: the toy of the chapter’s first code (NumPy seed 0): 10 responses, a reference policy that is the softmax of standard normal logits, rewards uniform on \([0,1]\), \(\beta=0.5\), and the KL-regularized optimum \(\pi^*\propto\pi_{\text{ref}}\,e^{r/\beta}\); the script draws and discards \(100{,}000\times16\) samples from \(\pi^*\) to keep its random stream aligned with the chapter’s code. A tabular softmax policy starts at the reference and takes 4,000 steps with learning rate 0.5 on groups of 256 sampled responses with the group mean as baseline, with the penalty \(\beta\log(\pi/\pi_{\text{ref}})\) subtracted from the reward (\(k_1\) in the reward), or with \(\beta\) times the sampled gradient of \(k_1=\log(\pi/\pi_{\text{ref}})\) or of \(k_3=\pi_{\text{ref}}/\pi-1-\log(\pi_{\text{ref}}/\pi)\) subtracted as a loss. Bars: the reference, \(\pi^*\), and the two loss versions, over the responses sorted by reward (the tick labels); instead of drawing the reward version, the script prints its total variation from \(\pi^*\). Right: 4 answers, the first correct (reward 1), each followed by up to 40 tokens, each of which ends the response with probability given by the sigmoid of a logit for that answer and position, starting at \(-2\); the answer logits start at 0. Groups of 8 responses; GRPO standardizes the rewards within the group and weights each response by \(1/(8\times\text{its length})\), and Dr. GRPO only centers them and weights each response by \(1/(8\times41)\); learning rates 2 (GRPO) and 20 (Dr. GRPO), a tenth of that for the answer logits, and 1,000 steps. Every 10 steps, the expected length of the correct answer’s responses and the mean over the incorrect ones, averaged over 20 runs (NumPy seeds 0–19).

29. Generalist agents, meta-RL, and open-endedness {#29-generalist-agents-meta-rl-and-open-endedness}

Script: ch29_meta.py.

Local figure	Data and construction
rl-meta.png	Left: the deceptive maze of the chapter’s second code, the square \([-1,1]^2\) with a cup of three walls, from \((-0.6,0.4)\) to \((0.6,0.4)\) and down from both ends to \(y=-0.3\), the start at the origin inside it, and the goal at \((0,0.8)\) above it. A genome is 12 two-dimensional moves, each shortened to length at most 0.3, stopped just before a wall, and clipped to the square. Populations of 50 start from \(\mathcal N(0,0.1^2)\) genomes; in each generation the best 10 by score are parents, drawn with replacement and mutated by \(\mathcal N(0,0.05^2)\) noise. The objective score is minus the distance to the goal, and the novelty score is the mean distance to the 10 nearest final positions in the archive and the population, with each individual added to the archive with probability 0.1. 60 generations (NumPy seed 0 for each search); the final positions of both populations and the novelty archive. Right: dependent two-armed Bernoulli bandits with means \((p,1-p)\), \(p\) uniform, and 30 pulls. RL²: a GRU with 32 hidden units whose input is the previous action (one-hot), the previous reward, and the fraction of pulls made, with linear policy and value heads, trained as in the chapter’s first code by an advantage actor-critic with discount 0.9 and returns divided by 5, standardized advantages, value loss weight 0.5, and an entropy bonus of 0.02 that decreases linearly to 0 at iteration 700; Adam at learning rate \(3\times10^{-3}\), gradient norm clipped at 1, and 1,000 iterations of 128 tasks (torch seed 0). On 20,000 new tasks (NumPy seed 0), the cumulative expected regret of UCB1 (bonus \(\sqrt{2\ln(t+1)/n}\), each arm once first), Thompson sampling with the posterior of \(p\) on a grid of 1,000 points and a uniform prior, and the RL² agent acting greedily, with its rewards from torch’s generator. The script prints the final regrets.

rl-meta.png

Left: the deceptive maze of the chapter’s second code, the square \([-1,1]^2\) with a cup of three walls, from \((-0.6,0.4)\) to \((0.6,0.4)\) and down from both ends to \(y=-0.3\), the start at the origin inside it, and the goal at \((0,0.8)\) above it. A genome is 12 two-dimensional moves, each shortened to length at most 0.3, stopped just before a wall, and clipped to the square. Populations of 50 start from \(\mathcal N(0,0.1^2)\) genomes; in each generation the best 10 by score are parents, drawn with replacement and mutated by \(\mathcal N(0,0.05^2)\) noise. The objective score is minus the distance to the goal, and the novelty score is the mean distance to the 10 nearest final positions in the archive and the population, with each individual added to the archive with probability 0.1. 60 generations (NumPy seed 0 for each search); the final positions of both populations and the novelty archive. Right: dependent two-armed Bernoulli bandits with means \((p,1-p)\), \(p\) uniform, and 30 pulls. RL²: a GRU with 32 hidden units whose input is the previous action (one-hot), the previous reward, and the fraction of pulls made, with linear policy and value heads, trained as in the chapter’s first code by an advantage actor-critic with discount 0.9 and returns divided by 5, standardized advantages, value loss weight 0.5, and an entropy bonus of 0.02 that decreases linearly to 0 at iteration 700; Adam at learning rate \(3\times10^{-3}\), gradient norm clipped at 1, and 1,000 iterations of 128 tasks (torch seed 0). On 20,000 new tasks (NumPy seed 0), the cumulative expected regret of UCB1 (bonus \(\sqrt{2\ln(t+1)/n}\), each arm once first), Thompson sampling with the posterior of \(p\) on a grid of 1,000 points and a uniform prior, and the RL² agent acting greedily, with its rewards from torch’s generator. The script prints the final regrets.

30. The theory of reinforcement learning {#30-the-theory-of-reinforcement-learning}

Script: ch30_theory.py.

Local figure	Data and construction
rl-theory.png	Left: the hard instance of the chapter’s first code, two actions that each pay 1 per step and continue with probability \(p=(4\gamma-1)/(3\gamma)\), so that the optimal value is \(1/(1-\gamma p)\), for \(\gamma=0.8\), 0.9, 0.95, 0.98, and 0.99, with 500 repetitions and all draws from one generator (NumPy seed 0). Model-based planning estimates each action’s \(p\) from 20,000 samples and returns \(1/(1-\gamma\max\hat p)\); Q-learning with a generative model makes 20,000 synchronous updates, each with one sample per action and step size \(1/(1+(1-\gamma)t)\). The mean absolute error of each estimate of the optimal value against the effective horizon \(1/(1-\gamma)\), on log–log axes, with reference lines of slopes 3/2 and 2 through the first points. Right: RiverSwim, as in the chapter’s second code, with 6 states, 2 actions, horizon 20, and 3,000 episodes from state 0. “Left” moves left, paying 0.005 in the leftmost state; “right” moves right with probability 0.35, stays with 0.6, and moves left with 0.05, except that it stays with 0.4 and moves right with 0.6 in the leftmost state, and stays with 0.6 and moves left with 0.4 in the rightmost, where it pays 1. The agents plan by finite-horizon backward induction on the empirical transitions (uniform for unvisited pairs) and the known rewards, with values capped at the number of remaining steps: ε-greedy (the greedy policy with weight 0.9 mixed with the uniform one), UCBVI with bonus \(0.2H/\sqrt n\), and PSRL, which samples each transition row from a Dirichlet distribution with parameters 1 plus the counts. The regret of each episode’s policy is computed exactly; the cumulative regret is averaged over 5 runs (NumPy seeds 0–4) and drawn on log–log axes with a reference line of slope 1/2. The script prints the fitted slopes of the left panel, the Q-learning slope without the longest horizon.

rl-theory.png

Left: the hard instance of the chapter’s first code, two actions that each pay 1 per step and continue with probability \(p=(4\gamma-1)/(3\gamma)\), so that the optimal value is \(1/(1-\gamma p)\), for \(\gamma=0.8\), 0.9, 0.95, 0.98, and 0.99, with 500 repetitions and all draws from one generator (NumPy seed 0). Model-based planning estimates each action’s \(p\) from 20,000 samples and returns \(1/(1-\gamma\max\hat p)\); Q-learning with a generative model makes 20,000 synchronous updates, each with one sample per action and step size \(1/(1+(1-\gamma)t)\). The mean absolute error of each estimate of the optimal value against the effective horizon \(1/(1-\gamma)\), on log–log axes, with reference lines of slopes 3/2 and 2 through the first points. Right: RiverSwim, as in the chapter’s second code, with 6 states, 2 actions, horizon 20, and 3,000 episodes from state 0. “Left” moves left, paying 0.005 in the leftmost state; “right” moves right with probability 0.35, stays with 0.6, and moves left with 0.05, except that it stays with 0.4 and moves right with 0.6 in the leftmost state, and stays with 0.6 and moves left with 0.4 in the rightmost, where it pays 1. The agents plan by finite-horizon backward induction on the empirical transitions (uniform for unvisited pairs) and the known rewards, with values capped at the number of remaining steps: ε-greedy (the greedy policy with weight 0.9 mixed with the uniform one), UCBVI with bonus \(0.2H/\sqrt n\), and PSRL, which samples each transition row from a Dirichlet distribution with parameters 1 plus the counts. The regret of each episode’s policy is computed exactly; the cumulative regret is averaged over 5 runs (NumPy seeds 0–4) and drawn on log–log axes with a reference line of slope 1/2. The script prints the fitted slopes of the left panel, the Q-learning slope without the longest horizon.

31. Reinforcement learning in the real world {#31-reinforcement-learning-in-the-real-world}

Script: ch31_realworld.py.

Local figure	Data and construction
rl-realworld.png	Left (no randomness): the constrained gridworld of the chapter’s third code, \(5\times7\) cells (rows counted from the top, from 0) with the start at row 2, column 0, an absorbing goal at row 2, column 6, reward \(-1\) per step, a hazard cost of 1 per step in row 2, columns 2–4, and a budget of 0.3 on the discounted cost, with \(\gamma=0.95\); each action moves in its direction with probability 0.85 and in each other direction with probability 0.05, staying in place at the edges. Lagrangian learning from the uniform policy with tabular softmax logits that grow by 0.2 times the exact action values of the reward \(r-\lambda c\) at each iteration, and the multiplier updated by gradient ascent, \(\lambda\leftarrow\max(0,\lambda+0.5e)\), or by a PI controller, \(\lambda=\max(0,5e+I)\) with \(I\leftarrow\max(0,I+0.5e)\), where \(e\) is the hazard cost minus 0.3; 1,500 iterations. The hazard cost of the current policy on an axis from 0.1 to 0.5, which cuts off the first iterations (they start at the uniform policy’s cost, 1.28, according to the docstring), with the budget dashed. Right: the pendulum of the chapter’s fourth code, \(\ddot\theta=10\sin\theta+u/m\) with Euler steps of 0.02 s and noise of \(0.02\sqrt{0.02}\) times a standard normal value on the angular velocity at each step, 200 steps from angles uniform on \([-0.2,0.2]\) at rest, and the torque clipped to \([-5,5]\) and applied after a delay of \(d\) steps; the cost is \(\theta^2+0.1\,\omega^2+10^{-4}u^2\) per step, and a fall past 1 rad costs the number of remaining steps, after which nothing more is added. Linear controllers \(u=-k_1\theta-k_2\omega\) come from a grid of 40 values of \(k_1\) from 10 to 400 and 30 of \(k_2\) from 1 to 60, the best being the one with the lowest mean cost over a set of rollouts. Nominal: best for \(m=1\) without delay (64 rollouts, NumPy seed 0). Randomized: best over 256 rollouts with \(m\) uniform on \([0.5,2]\) and delays of 0–3 steps (NumPy seed 1 for these draws and for the rollouts). Adaptive: the randomized gains for 5 steps, then the best gains of the most likely of 28 candidate models (7 masses geometrically spaced from 0.5 to 2 and delays of 0–3 steps), ranked by the squared error of their predictions of the angular velocity. Oracle: best for the true mass and delay. The mean cost of 400 test rollouts (NumPy seed 7) with \(m=0.5\) and delays of 0–5 steps (0–100 ms), on a logarithmic axis, with the training range up to 60 ms shaded. The script prints the costs.

rl-realworld.png

Left (no randomness): the constrained gridworld of the chapter’s third code, \(5\times7\) cells (rows counted from the top, from 0) with the start at row 2, column 0, an absorbing goal at row 2, column 6, reward \(-1\) per step, a hazard cost of 1 per step in row 2, columns 2–4, and a budget of 0.3 on the discounted cost, with \(\gamma=0.95\); each action moves in its direction with probability 0.85 and in each other direction with probability 0.05, staying in place at the edges. Lagrangian learning from the uniform policy with tabular softmax logits that grow by 0.2 times the exact action values of the reward \(r-\lambda c\) at each iteration, and the multiplier updated by gradient ascent, \(\lambda\leftarrow\max(0,\lambda+0.5e)\), or by a PI controller, \(\lambda=\max(0,5e+I)\) with \(I\leftarrow\max(0,I+0.5e)\), where \(e\) is the hazard cost minus 0.3; 1,500 iterations. The hazard cost of the current policy on an axis from 0.1 to 0.5, which cuts off the first iterations (they start at the uniform policy’s cost, 1.28, according to the docstring), with the budget dashed. Right: the pendulum of the chapter’s fourth code, \(\ddot\theta=10\sin\theta+u/m\) with Euler steps of 0.02 s and noise of \(0.02\sqrt{0.02}\) times a standard normal value on the angular velocity at each step, 200 steps from angles uniform on \([-0.2,0.2]\) at rest, and the torque clipped to \([-5,5]\) and applied after a delay of \(d\) steps; the cost is \(\theta^2+0.1\,\omega^2+10^{-4}u^2\) per step, and a fall past 1 rad costs the number of remaining steps, after which nothing more is added. Linear controllers \(u=-k_1\theta-k_2\omega\) come from a grid of 40 values of \(k_1\) from 10 to 400 and 30 of \(k_2\) from 1 to 60, the best being the one with the lowest mean cost over a set of rollouts. Nominal: best for \(m=1\) without delay (64 rollouts, NumPy seed 0). Randomized: best over 256 rollouts with \(m\) uniform on \([0.5,2]\) and delays of 0–3 steps (NumPy seed 1 for these draws and for the rollouts). Adaptive: the randomized gains for 5 steps, then the best gains of the most likely of 28 candidate models (7 masses geometrically spaced from 0.5 to 2 and delays of 0–3 steps), ranked by the squared error of their predictions of the angular velocity. Oracle: best for the true mass and delay. The mean cost of 400 test rollouts (NumPy seed 7) with \(m=0.5\) and delays of 0–5 steps (0–100 ms), on a logarithmic axis, with the training range up to 60 ms shaded. The script prints the costs.

Shared style

rlfig.py selects Matplotlib’s Agg backend, and its setup() sets the font DejaVu Sans (10.5 pt, titles 11.5 pt, DejaVu Sans math text), dark axes of line width 0.8 without top and right spines, dark ticks, frameless legends, and white figure, axes, and saved backgrounds. Its save() writes each figure to Sources/Images as a 200 dpi PNG with a tight bounding box and 0.08 inches of padding, closes it, and prints its name; the scripts of chapters 27–31 pass names without the extension, to which Matplotlib adds .png. The palette is BLUE #2267a5, ORANGE #d46a23, GREEN #39805a, PURPLE #995aa4, RED #b32b35, DARK #152330 for axes and ticks, GRAY #586473, and LIGHT #c1c8cf, with light tints of the first five colors defined for fills; the default color cycle is blue, orange, green, purple, red, and gray. Individual scripts set smaller font sizes for titles, annotations, and legends, and use Matplotlib’s RdBu, Blues, viridis, and magma colormaps and two-color and three-color colormaps built from the tints.