31. Reinforcement Learning in the Real World

Specifying the problem

Rewards are designed

Every chapter so far began with an MDP. Outside a textbook, someone has to write it down: decide what the agent observes, what it may do and how often, what it is rewarded for, and what it must never do. Of these, the reward is the hardest, because it has to say what the designer wants in a form an optimizer cannot misread. The reward hypothesis of chapter 1 says that any goal can be expressed as a reward; it says nothing about how easy the right reward is to find.

Designers face two opposite failures. A sparse reward, 1 for success and 0 otherwise, says exactly what is wanted, but an agent may almost never receive it, and learning then waits on exploration (chapter 22). A dense reward, a hand-made score for progress, gives a signal at every step, but it encodes the designer’s guesses about how to succeed, and the agent follows those guesses wherever they lead. An agent learning to ride a bicycle to a goal, rewarded for progress toward it and not penalized for riding away, learned to ride in small circles near its starting point, collecting the reward for the approaching half of every circle (Randløv and Alstrøm, 1998, as described by Ng, Harada, and Russell, 1999). A boat-racing agent rewarded for the game’s score found a lagoon where it could circle forever, knocking over three respawning targets, catching fire and crashing, and scored on average 20% more than human players without finishing the race (Clark and Amodei, 2016). In a simulated block-stacking task, a shaping term for grasping, computed from the wrong reference point on the brick, taught the arm to flip the brick over instead (Popov et al., 2017). Collections of such cases now run to dozens (Krakovna et al., 2020), and they share a structure: the agent did what it was rewarded for, and the reward differed from the intent in a place the designer had not thought about.

Practice has converged on a few habits. Reward the outcome rather than the method when exploration allows it, and add dense terms only where they are needed. Keep regularizing terms, for energy, smoothness, or wear, small and bounded, so that they cannot dominate the objective. Watch the learned behavior, not only the return curve, since a rising return is exactly what reward hacking looks like. And treat reward design as an iterative loop, in which the designer trains, inspects, and revises; systems such as Eureka now automate that loop with a language model writing the reward code (chapter 29), and learned rewards, from demonstrations (chapter 25) or from preferences (chapter 28), replace hand-writing by fitting.

Potential-based shaping

One family of dense rewards is safe by construction. Potential-based shaping adds to the reward of each transition

\[F(s,a,s')=\gamma\Phi(s')-\Phi(s)\]

for a potential \(\Phi\) defined on states, with \(\Phi=0\) at terminal states. Along any trajectory the added terms telescope: the shaped return from \(s_0\) differs from the original by \(-\Phi(s_0)\), whatever the agent does afterward, so every action value shifts by the same amount, \(Q'^{\pi}(s,a)=Q^\pi(s,a)-\Phi(s)\), and the optimal policies of the two problems coincide. Ng, Harada, and Russell (1999) proved the converse as well: if \(F\) is not of this form, there are dynamics and rewards for which it changes the optimal policy (appendix A). The potential can be any function of the state, and a good one approximates the optimal value: with \(\Phi=V^*\), the expected shaped reward of an action is its advantage \(A^*(s,a)\), zero for optimal actions and negative for the others, and the problem becomes one a greedy, myopic agent can solve (exercise 31.2). The same idea extends to potentials that depend on the action (Wiewiora, Cottrell, and Elkan, 2003) and to potentials that change over time (Devlin and Kudenko, 2012).

The guarantee is about the optimum, not about learning. Wiewiora (2003) showed that Q-learning with potential-based shaping, started from zero, makes exactly the same decisions as Q-learning without shaping started from \(Q_0(s,a)=\Phi(s)\): shaping is an initialization of the value estimates. A potential therefore helps as much as a good initial guess of the values would, and it can mislead as much as a bad one. The next code measures both effects, and the damage done by a progress bonus that is not potential-based, in a gridworld whose only reward is at the goal.

import numpy as np

# Reward shaping in a sparse gridworld. A 15 x 15 grid, start at (0, 0), goal at (14, 14), reward 1 on reaching the
# goal and 0 otherwise, gamma = 0.98, four moves (a move into the edge stays put), at least 28 moves to the goal.
# Q-learning (step size 0.5, epsilon-greedy with epsilon 0.1, ties broken at random, Q initialized to 0) runs for 300
# episodes of at most 1,000 steps, 10 seeds, with the sparse reward plus a shaping term F(s, s'):
#   potential-based, F = gamma Phi(s') - Phi(s) (Phi = 0 at the goal), with d(s) the Manhattan distance to the goal:
#     Phi = 1 - d/28, a rough guess of the value; Phi = -d/28, the same guess shifted by -1; Phi = +d/28, backward;
#   a progress bonus, F = 0.1 for every move that reduces d, and 0 for every move that increases it.
# We count the steps used in training and follow the final greedy policy, which is judged by the true reward only.
N, GOAL, GAMMA = 15, (14, 14), 0.98
MOVES = ((1, 0), (-1, 0), (0, 1), (0, -1))
dist = lambda s: abs(GOAL[0] - s[0]) + abs(GOAL[1] - s[1])
SHAPING = {
    "none": lambda s, s2: 0.0,
    "Phi = 1 - d/28": lambda s, s2: (0 if s2 == GOAL else GAMMA * (1 - dist(s2) / 28)) - (1 - dist(s) / 28),
    "Phi = -d/28": lambda s, s2: (0 if s2 == GOAL else GAMMA * -dist(s2) / 28) - -dist(s) / 28,
    "Phi = +d/28": lambda s, s2: (0 if s2 == GOAL else GAMMA * dist(s2) / 28) - dist(s) / 28,
    "progress bonus": lambda s, s2: 0.1 if dist(s2) < dist(s) else 0.0,
}


def step(s, a):
    return min(max(s[0] + MOVES[a][0], 0), N - 1), min(max(s[1] + MOVES[a][1], 0), N - 1)


def train(F, seed, episodes=300):
    rng = np.random.default_rng(seed)
    Q, used = np.zeros((N, N, 4)), 0
    for ep in range(episodes):
        s = (0, 0)
        for t in range(1000):
            a = rng.integers(4) if rng.random() < 0.1 else rng.choice(np.flatnonzero(Q[s] == Q[s].max()))
            s2 = step(s, a)
            r = (s2 == GOAL) + F(s, s2)
            Q[s][a] += 0.5 * ((r if s2 == GOAL else r + GAMMA * Q[s2].max()) - Q[s][a])
            s, used = s2, used + 1
            if s == GOAL:
                break
    s, t = (0, 0), 0                                      # the greedy policy, run for up to 1,000 steps
    while s != GOAL and t < 1000:
        s, t = step(s, Q[s].argmax()), t + 1
    return used, t if s == GOAL else None


print("                    steps in training        greedy policy        its length")
print("  shaping           median     largest      reaches the goal    (optimal 28)")
for name, F in SHAPING.items():
    res = [train(F, seed) for seed in range(10)]
    used, ok = [u for u, _ in res], [L for _, L in res if L is not None]
    print(f"  {name:16s}{np.median(used):9,.0f}{max(used):12,d}{len(ok):16d} of 10" + (f"{np.mean(ok):16.1f}" if ok else f"{'never':>16s}"))
#                     steps in training        greedy policy        its length
#   shaping           median     largest      reaches the goal    (optimal 28)
#   none               34,345      41,361              10 of 10            28.2
#   Phi = 1 - d/28      9,746       9,888              10 of 10            28.0
#   Phi = -d/28        52,871     274,945               9 of 10            28.7
#   Phi = +d/28       173,811     195,787               9 of 10            28.0
#   progress bonus     53,780      60,659               0 of 10           never

Without shaping, Q-learning spends a median of about 34,000 steps over its 300 episodes, most of them in the first few dozen episodes, while it still wanders before reaching the goal. The potential \(\Phi=1-d/28\), a rough guess of the value that decreases with the Manhattan distance \(d\) to the goal, cuts that to under 10,000 and finds the shortest path in every run. The same potential lowered by 1 describes the same differences between states, and it still leaves the optimal policy unchanged, yet it made learning slower than no shaping at all and erratic, with one run using 275,000 steps: a negative potential is a pessimistic initialization, and against pessimistic estimates, a move into a wall that leaves the agent where it is looks better than a move into an unexplored state (exercise 31.2). A potential that points away from the goal is five times slower than none, but because it is potential-based, it leaves the optimal policy unchanged, and nine of its ten runs end with shortest paths. The progress bonus is worse than slow. Stepping toward the goal and back earns 0.1 every two steps forever, worth more than entering the goal, so in the shaped problem the optimal behavior is to hover next to the goal and never reach it (exercise 31.1): no run’s greedy policy reaches the goal, exactly the bicycle’s circles.

When the proxy is optimized

A reward that is only approximately right is a proxy, and optimizing a proxy hard enough tends to find the places where it and the intent diverge, an instance of Goodhart’s law. Skalse et al. (2022) made this precise. They call a proxy unhackable with respect to a true reward if no change of policy can increase the proxy’s return while decreasing the true return, and showed that over the set of all stochastic policies, two rewards are unhackable with respect to each other only if one of them is constant: every nontrivial proxy can be gamed by some policy. Nontrivial unhackable pairs exist only for special sets of policies, such as finite sets or deterministic policies; any set containing an open set of stochastic policies, even a small neighborhood of a trusted policy, still admits hacking. Limiting how far optimization moves from a trusted policy, as the KL penalties of chapter 28 do, therefore cannot rule hacking out, but it limits how far the true return can fall, which is why it is a real safeguard and not only a regularizer. Learned rewards are proxies by construction, and their overoptimization, the true quality peaking and then falling as the proxy keeps rising, is measured in NLP chapter 11; reward hacking and the questions it raises about overseeing capable optimizers are central themes of the Safety and Frontier module.

Many real objectives also have several parts that trade off: speed and energy, throughput and wear, performance and risk. Folding them into one reward with fixed weights is common, but the weights have no meaning an operator can check, and they must be retuned whenever the scales change. When one of the parts is really a requirement, a bound on an expected cost that must not be exceeded, a constrained formulation states it directly (below); when the trade-off itself is to be explored, multi-objective RL learns a set of policies along the Pareto front (Roijers, Vamplew, Whiteson, and Dazeley, 2013).

Sparse rewards and goals

Goal-conditioned learning and hindsight

Many tasks are naturally about reaching a goal: a position, a configuration of objects, an arrangement of bits. A goal-conditioned policy \(\pi(a\mid s,g)\) and value \(Q(s,a,g)\), trained over a distribution of goals, generalize across goals as they do across states, the universal value functions of chapter 29, and the reward is usually sparse: \(r(s',g)=0\) if \(s'\) achieves \(g\), and \(-1\) otherwise. Goal conditioning offers something ordinary RL does not. When the reward is a known function of the state and the goal, one experience can be scored against any goal, an observation that goes back to Kaelbling (1993), whose agent updated its values for all goals from every transition.

Hindsight experience replay (HER; Andrychowicz et al., 2017) turns the observation into a replay strategy for off-policy learners. A trajectory that failed to reach its goal \(g\) did reach the states it visited, so each transition \((s_t,a_t,s_{t+1})\) is stored not only with \(g\) but also with a few goals achieved later in the same episode, with the reward recomputed: for those goals, part of the trajectory is a success. The “future” strategy with four relabeled goals per transition worked best. The authors’ test case is bit flipping: the state and the goal are strings of \(n\) bits, each action flips one bit, and the reward is \(-1\) until the state equals the goal. A random start and goal differ in about \(n/2\) bits, and random exploration essentially never matches all \(n\), so an agent that learns only from its real goals never sees a reward. The next code reproduces the comparison at a small scale.

import numpy as np
import torch
import torch.nn as nn

# Hindsight experience replay on bit flipping (Andrychowicz et al., 2017). The state and the goal are strings of n
# bits; each action flips one bit; the reward is 0 on reaching the goal, which ends the episode, and -1 otherwise;
# an episode lasts at most n steps. A random start and goal differ in about n/2 bits, and random flipping almost
# never matches all n. DQN (MLP 2n -> 256 -> n, epsilon 0.2, gamma 0.98, soft target updates) is trained for
# 2,400 episodes, 16 per cycle, each followed by 40 minibatch updates, with and without hindsight relabeling: every
# transition is also stored with 4 goals replaced by states reached later in the same episode ("future" strategy).
torch.set_num_threads(1)


def train(n, her, seed, cycles=150, E=16, gamma=0.98):
    rng = np.random.default_rng(seed); torch.manual_seed(seed)
    make = lambda: nn.Sequential(nn.Linear(2 * n, 256), nn.ReLU(), nn.Linear(256, n))
    net, tgt = make(), make(); tgt.load_state_dict(net.state_dict())
    opt = torch.optim.Adam(net.parameters(), lr=1e-3)
    q = lambda m, s, g: m(torch.as_tensor(np.concatenate([s, g], -1), dtype=torch.float32))
    buf = {k: [] for k in "sagn"}                  # state, action, goal, next state
    for c in range(cycles):
        s, g = rng.integers(0, 2, (E, n)), rng.integers(0, 2, (E, n))
        traj, acts, length = [s], [], np.full(E, n)
        for t in range(n):
            with torch.no_grad():
                a = np.where(rng.random(E) < 0.2, rng.integers(0, n, E), q(net, s, g).argmax(1).numpy())
            s = s.copy(); s[np.arange(E), a] ^= 1
            traj.append(s); acts.append(a)
            length = np.where((length == n) & (s == g).all(1) & (t < length), t + 1, length)
        traj, acts = np.stack(traj, 1), np.stack(acts, 1)       # (E, n+1, n), (E, n)
        e, t = np.nonzero(np.arange(n)[None] < length[:, None])  # the transitions actually taken
        goals = [g[e]]
        if her:                                                  # 4 goals achieved later in the same episode
            goals += [traj[e, rng.integers(t + 1, length[e] + 1)] for _ in range(4)]
        for gl in goals:
            buf["s"].append(traj[e, t]); buf["a"].append(acts[e, t]); buf["g"].append(gl); buf["n"].append(traj[e, t + 1])
        S, A, G, S2 = (np.concatenate(buf[k]) for k in "sagn")
        for _ in range(40):
            i = rng.integers(0, len(S), 128)
            done = torch.as_tensor((S2[i] == G[i]).all(1), dtype=torch.float32)
            with torch.no_grad():
                y = (done - 1 + gamma * (1 - done) * q(tgt, S2[i], G[i]).max(1).values).clamp(-1 / (1 - gamma), 0)
            loss = ((q(net, S[i], G[i]).gather(1, torch.as_tensor(A[i])[:, None])[:, 0] - y) ** 2).mean()
            opt.zero_grad(); loss.backward(); opt.step()
        with torch.no_grad():
            for p, pt in zip(net.parameters(), tgt.parameters()):
                pt.mul_(0.95).add_(0.05 * p)
    s, g = rng.integers(0, 2, (1000, n)), rng.integers(0, 2, (1000, n))  # greedy test on 1,000 new goals
    done = (s == g).all(1)
    for t in range(n):
        with torch.no_grad():
            a = q(net, s, g).argmax(1).numpy()
        s = s.copy(); s[np.arange(1000), a] ^= ~done; done |= (s == g).all(1)
    return done.mean(), (S2 == G).all(1).mean()


print("         greedy success on new goals    stored transitions that reach their goal     (mean of 2 seeds)")
print("  bits      DQN      DQN + HER                DQN      DQN + HER")
for n in (6, 10, 14, 18):
    dqn, her = (np.mean([train(n, h, seed) for seed in range(2)], 0) for h in (False, True))
    print(f"  {n:4d}{dqn[0]:9.2f}{her[0]:15.2f}{dqn[1]:19.4f}{her[1]:15.4f}")
#          greedy success on new goals    stored transitions that reach their goal     (mean of 2 seeds)
#   bits      DQN      DQN + HER                DQN      DQN + HER
#      6     1.00           1.00             0.1490         0.4726
#     10     0.01           1.00             0.0013         0.3438
#     14     0.00           1.00             0.0000         0.2780
#     18     0.00           0.84             0.0000         0.2429

DQN alone learns 6 bits, where 15% of its stored transitions happen to reach their goals, and fails from 10 bits on, where almost none do: it has nothing to learn from. With hindsight, about a quarter to a half of the stored transitions are successes for some goal at every size, and the same network, with the same budget, reaches every new goal at 10 and 14 bits and 84% of them at 18. With more training, the original paper found DQN limited to \(n\le13\) and HER solving \(n=50\). On simulated robot arms, HER learned pushing, sliding, and pick-and-place from binary rewards, where the shaped rewards the authors tried failed on every task, and a pick-and-place policy trained in simulation ran on a physical Fetch robot, succeeding in 2 of 5 trials at first and in 5 of 5 once observation noise was added in training, a small instance of the sim-to-real methods of a later section.

Hindsight beyond goals

Relabeling rewrites the goal after seeing the outcome, and in a stochastic environment that is a biased sample. A transition stored with the goal it happened to reach is always a success for that goal, so an action that reaches a state only by luck looks, in the relabeled data, like one that reaches it reliably, and value estimates become optimistic about risky actions (exercise 31.3). The bias vanishes in deterministic environments, and it can be corrected by importance weighting (Schramm et al., 2022) or offset by rewarding hindsight transitions differently from real ones (Lanka and Wu, 2018).

The same idea supports methods without value functions. Goal-conditioned supervised learning (GCSL; Ghosh et al., 2021) relabels each of the agent’s own trajectories with the goals it reached and trains the policy to imitate the relabeled actions, which are by definition good ways of reaching those goals; iterated, this improves the policy without rewards or critics, and is the goal-reaching relative of the return-conditioned policies of chapter 26. Relabeling also generalizes beyond goals: a trajectory can be relabeled with any task, reward function, or instruction for which it is good evidence, a choice that can be made by inverse RL (Eysenbach, Geng, Levine, and Salakhutdinov, 2020; Li, Pinto, and Abbeel, 2020). And the goals themselves can be generated: a curriculum of goals of intermediate difficulty, produced by a generative adversarial network (Florensa, Held, Geng, and Abbeel, 2018), or goals imagined by sampling from a generative model of observations, which let a robot practice reaching images of states it could produce itself (Nair et al., 2018).

Temporal abstraction

Options and semi-MDPs

People plan in steps of very different lengths: drive to the airport, then take the flight, and only at the lowest level move a foot or a wheel. An agent that chooses a primitive action every few milliseconds faces long horizons, slow credit assignment, and exploration that rarely strays far from where it started. Options (Sutton, Precup, and Singh, 1999) give it temporally extended actions. An option \(o=(\mathcal I_o,\pi_o,\beta_o)\) has an initiation set of states where it may start, a policy it follows while running, and a termination condition \(\beta_o(s)\), the probability of stopping in each state. Primitive actions are options that last one step. Choosing among options from the states where one ends is a semi-MDP, a decision process whose actions take variable amounts of time, and its values satisfy a Bellman equation in which an option that runs for \(k\) steps is followed by a discount of \(\gamma^k\):

\[Q(s,o)=\mathbb E\bigl[r_{t+1}+\gamma r_{t+2}+\dots+\gamma^{k-1}r_{t+k}+\gamma^k\max_{o'}Q(s_{t+k},o')\;\big|\;s_t=s,\ o\bigr],\]

where the expectation is over the random duration \(k\) as well as the rewards and the state where the option ends. Q-learning carries over directly, with the discounted reward accumulated during the option as the reward and \(\gamma^k\) as the discount (Bradtke and Duff, 1994). Because the options’ policies are known, experience can do more work: intra-option learning updates the value of every option consistent with each primitive step, and an option can be interrupted whenever switching to another looks better.

What options buy is reach. The value of a distant reward propagates back through a few option-level decisions instead of hundreds of primitive ones (exercise 31.6), and an agent exploring at the level of options travels far with each choice. What they cost is optimality: an agent restricted to a set of options can do no better than the best combination of them, which may be worse than the best primitive policy. Earlier formulations imposed hierarchies from above: feudal RL (Dayan and Hinton, 1992), with managers that set subgoals for sub-managers without caring how they are met; hierarchies of abstract machines (Parr and Russell, 1997), partial programs whose choice points are learned; and MAXQ (Dietterich, 2000), which decomposes the value function along a task hierarchy and distinguishes hierarchically optimal policies, the best a hierarchy allows, from recursively optimal ones, in which each subtask is solved in isolation.

Learning hierarchies

Where good options come from is the hard part. Early methods looked for bottleneck states, such as doorways, that lie on many successful trajectories, and made reaching them into subgoals (McGovern and Barto, 2001); eigenoptions (Machado, Bellemare, and Bowling, 2017) use the eigenvectors of the graph Laplacian of the state space as intrinsic rewards, each yielding an option that travels along one of the directions in which the space is hardest to cross. The option-critic architecture (Bacon, Harb, and Precup, 2017) learns the options’ policies and termination functions end to end by policy gradients, though without further pressure learned options tend to degenerate into one option that does everything or into options that switch at every step.

The approach that has scaled best is to make the high level choose goals for a goal-conditioned low level, with the low level rewarded intrinsically for reaching them: h-DQN (Kulkarni, Narasimhan, Saeedi, and Tenenbaum, 2016) with hand-specified subgoal objects; FeUdal networks (Vezhnevets et al., 2017), whose manager emits directions in a learned latent space at a slower time scale; HIRO (Nachum, Gu, Lee, and Levine, 2018), whose goals are desired changes of the state and which makes off-policy learning at the high level possible by relabeling old high-level actions with the goals that would best explain what the changing low level actually did; hierarchical actor-critic (Levy, Konidaris, Platt, and Saenko, 2019), which applies hindsight relabeling at every level; and Director (Hafner, Lee, Fischer, and Abbeel, 2022), which chooses goals in the latent space of a world model and learns both levels in imagination.

Why hierarchy helps has been examined as carefully as whether it does. Comparing hierarchical agents with flat agents given the same advantages one at a time, Nachum et al. (2019) attributed most of the benefit to better exploration, temporally extended and directed, and little to easier learning of the policy itself; flat agents that explore with temporally correlated, goal-directed behavior performed competitively with hierarchical ones. In current systems, hierarchy is often supplied rather than learned: a language model decomposes an instruction into steps that learned skills can execute, as in SayCan (Ahn et al., 2022), and a robot’s learned locomotion controller serves as the low level for navigation or manipulation planners. Temporal abstraction also has a humbler, universal form. The control frequency and the action repeat, the number of simulator steps each decision is held for, set the time scale of the problem, and choosing them is part of problem design (below).

Safety

Constrained MDPs

Many requirements are naturally bounds. A robot should reach its goal quickly and spend little time near people; a building controller should save energy and keep temperatures within limits; a trading agent should earn and keep its risk below a threshold. A constrained MDP (Altman, 1999) adds cost functions \(c_1,\dots,c_k\) and budgets \(d_1,\dots,d_k\) to an MDP and asks for

\[\max_\pi J_r(\pi)\quad\text{subject to}\quad J_{c_i}(\pi)\le d_i,\ i=1,\dots,k,\]

where \(J_r\) and \(J_{c_i}\) are the expected discounted totals of the reward and of each cost from the start. Budgets have units an operator can state and check, and they do not need retuning when the reward changes scale, which fixed penalty weights do.

The theory is that of linear programming. Expected discounted totals are linear in the discounted occupancy measure \(x(s,a)\) of the policy (chapter 1), and the occupancy measures of all policies are exactly the nonnegative solutions of a set of linear flow equations, so a constrained MDP is the linear program of chapter 2 with \(k\) more inequality constraints. Two consequences follow (appendix B). First, an optimal policy may have to randomize, but there is always one that randomizes in at most \(k\) states; with one constraint, in at most one. A fixed penalty turns the problem back into an ordinary MDP, whose optimal policies can be taken deterministic, so it cannot in general produce the constrained optimum. Second, strong duality holds: with the Lagrangian

\[L(\pi,\lambda)=J_r(\pi)-\sum_i\lambda_i\bigl(J_{c_i}(\pi)-d_i\bigr),\qquad \lambda_i\ge0,\]

the constrained optimum equals \(\min_{\lambda\ge0}\max_\pi L(\pi,\lambda)\), and the inner maximization is an ordinary MDP with the penalized reward \(r-\sum_i\lambda_ic_i\). Paternain, Chamon, Calvo-Fullana, and Ribeiro (2019) showed that the duality gap is zero despite the nonconvexity of the problem in the policy, and that for parameterized policies it is bounded by a multiple of the parameterization’s approximation error, so nearly zero for rich ones, the justification of the Lagrangian methods that dominate practice: an RL algorithm improves the policy on the penalized reward while the multipliers rise when a constraint is violated and fall when it is slack. This is RCPO (Tessler, Mankowitz, and Mannor, 2019) and the PPO-Lagrangian baselines of the Safety Gym benchmark (Ray, Achiam, and Amodei, 2019). Constrained policy optimization (CPO; Achiam, Held, Tamar, and Abbeel, 2017) instead adds a linearized cost constraint to each trust-region step of chapter 20, aiming to satisfy the constraint approximately at every update rather than on average.

Lagrangian learning has a characteristic failure: the multiplier is a slow feedback loop around a lagging system. The multiplier grows while the policy violates the constraint, the policy responds only after the multiplier has overshot, and the cost swings from violation to excessive caution and back. Stooke, Achiam, and Abbeel (2020) observed that the usual multiplier update, \(\lambda\leftarrow\max(0,\lambda+\eta(J_c-d))\), is integral control of the constraint violation, and that adding the proportional and derivative terms of a PID controller damps the oscillation. The next code shows the whole picture on a small gridworld, solving the constrained problem exactly and then by Lagrangian learning with each update of the multiplier.

import numpy as np
from scipy.optimize import linprog

# A constrained MDP. A robot crosses a 5 x 7 grid from (2, 0) to (2, 6); the three cells between, (2, 2) to (2, 4),
# are hazardous. Each step costs reward -1, each step taken in a hazard costs 1 unit of the constraint cost, and moves
# slip in a random direction with probability 0.2; gamma = 0.95. The objective is the discounted return, subject to
# a discounted hazard cost of at most 0.3. We compare: fixed penalties (optimal policies for r - lambda c, found by
# value iteration); the exact constrained optimum, a linear program over occupancy measures; and Lagrangian learning,
# in which a softmax policy takes exact natural-gradient steps on r - lambda c while lambda follows the constraint
# violation, either by gradient ascent (lambda += 0.5 (cost - 0.3)) or by a PI controller that adds a proportional
# term (lambda = 5 (cost - 0.3) + sum of 0.5 (cost - 0.3)), as in Stooke, Achiam, and Abbeel (2020).
H, W, GAMMA, BUDGET = 5, 7, 0.95, 0.3
S, A, MOVES = H * W, 4, ((-1, 0), (1, 0), (0, -1), (0, 1))
P, R, C = np.zeros((S, A, S)), np.full((S, A), -1.0), np.zeros((S, A))
for y in range(H):
    for x in range(W):
        s = y * W + x
        if (y, x) == (2, 6):                                    # the goal is absorbing and free
            P[s, :, s], R[s] = 1, 0
            continue
        C[s] = (y, x) in ((2, 2), (2, 3), (2, 4))
        for a in range(A):
            for b, (dy, dx) in enumerate(MOVES):
                P[s, a, min(max(y + dy, 0), H - 1) * W + min(max(x + dx, 0), W - 1)] += 0.8 * (a == b) + 0.05
mu0 = np.eye(S)[2 * W]


def values(pi, f):
    """Per-state discounted totals of f(s, a) under policy pi, and the total from the start."""
    v = np.linalg.solve(np.eye(S) - GAMMA * np.einsum("sa,sat->st", pi, P), (pi * f).sum(1))
    return v, v @ mu0


def optimal(lam):
    Q = np.zeros((S, A))
    for _ in range(1000):
        Q = R - lam * C + GAMMA * P @ Q.max(1)
    return np.eye(A)[Q.argmax(1)]


print("fixed penalty    return    cost")
for lam in (0, 0.8, 1, 3, 30):
    pi = optimal(lam)
    print(f"  lambda = {lam:<4}{values(pi, R)[1]:8.3f}{values(pi, C)[1]:8.3f}")

# the exact optimum: maximize sum x r subject to the flow constraints and sum x c <= 0.3, over occupancies x >= 0
flow = np.kron(np.eye(S), np.ones(A)) - GAMMA * P.reshape(S * A, S).T
lp = linprog(-R.ravel(), A_ub=C.ravel()[None], b_ub=[BUDGET], A_eq=flow, b_eq=mu0, method="highs")
x = lp.x.reshape(S, A); visited = x.sum(1) > 1e-9
mixed = (x[visited].max(1) < x[visited].sum(1) - 1e-9).sum()
print(f"linear program   {-lp.fun:.3f}   {C.ravel() @ lp.x:.3f}   randomizes in {mixed} state(s); multiplier {-lp.ineqlin.marginals[0]:.3f}")


def lagrangian(controller, iters=3000):
    logits, lam, integral, hist = np.zeros((S, A)), 0.0, 0.0, []
    for k in range(iters):
        pi = np.exp(logits - logits.max(1, keepdims=True)); pi /= pi.sum(1, keepdims=True)
        J_r, J_c = values(pi, R)[1], values(pi, C)[1]
        hist.append((J_r, J_c, lam))
        err = J_c - BUDGET
        if controller == "gradient ascent":
            lam = max(0.0, lam + 0.5 * err)
        else:
            integral = max(0.0, integral + 0.5 * err); lam = max(0.0, 5 * err + integral)
        f = R - lam * C
        logits += 0.2 * (f + GAMMA * P @ values(pi, f)[0])     # natural gradient step: logits += eta Q_lambda
    return np.array(hist)


print("Lagrangian       final: return   cost   lambda   iterations over budget   total violation   last 1,000: mean return, cost")
for controller in ("gradient ascent", "PI controller"):
    h = lagrangian(controller)
    over = np.maximum(h[:, 1] - BUDGET, 0)
    print(f"  {controller:16s}{h[-1, 0]:12.3f}{h[-1, 1]:7.3f}{h[-1, 2]:9.3f}{(over > 0.005).mean():21.0%}{over.sum():18.1f}"
          f"{h[-1000:, 0].mean():17.3f}{h[-1000:, 1].mean():7.3f}")
# fixed penalty    return    cost
#   lambda = 0     -6.712   2.087
#   lambda = 0.8   -7.819   0.330
#   lambda = 1     -7.937   0.207
#   lambda = 3     -8.090   0.126
#   lambda = 30    -9.256   0.013
# linear program   -7.848   0.300   randomizes in 1 state(s); multiplier 0.962
# Lagrangian       final: return   cost   lambda   iterations over budget   total violation   last 1,000: mean return, cost
#   gradient ascent       -7.820  0.329    0.794                  60%              48.2           -7.847  0.301
#   PI controller         -7.848  0.300    0.962                   5%               3.1           -7.848  0.300

Without the constraint, the shortest route crosses the hazards, with a hazard cost of 2.09. Every fixed penalty yields a deterministic policy, and as \(\lambda\) grows the cost falls in jumps: to 0.330 for \(\lambda\) between about 0.63 and 0.96, just over the budget of 0.3, then to 0.207, and further for larger penalties. No penalty spends the budget exactly, and the best policy a penalty yields that satisfies it, for any \(\lambda\) between 0.962 and 1.835 such as \(\lambda=1\), returns \(-7.937\). The linear program does better, \(-7.848\), by randomizing in a single state between the two policies on either side of the budget, those with costs 0.330 and 0.207, in the proportion that spends the budget exactly; its multiplier, 0.962, is the penalty at which those two policies tie. Lagrangian learning with gradient ascent on the multiplier finds this solution only on average: its iterates, averaged over the last 1,000 iterations, have the optimal return and cost, but the current policy keeps swinging between the two tied policies, its cost cycling between about 0.24 and 0.33, over budget in 60% of the iterations and at the end. The PI controller settles on the randomized optimum and the exact multiplier, and its total violation over training is about a fifteenth of gradient ascent’s. The left panel of the figure after the fourth code shows the two trajectories. Methods with guarantees for the last iterate rather than the average, such as optimistic gradient updates (Moskovitz et al., 2023), address the same problem from the side of optimization theory.

Safe exploration

A constraint satisfied in expectation, after training, says nothing about what happens during training, and an agent cannot learn that an action is dangerous without trying it, being told, or predicting it. Safe exploration therefore always rests on prior knowledge, and methods differ in its form (García and Fernández, 2015; Brunke et al., 2022).

• Shields and safety filters. A separate component checks each proposed action and replaces it if it could lead to an unsafe state. The check can come from a formal specification in temporal logic and an abstraction of the dynamics (Alshiekh et al., 2018), from a control barrier function, a function of the state that must not decrease past zero and that turns safety into a constraint on each action (Cheng, Orosz, Murray, and Burdick, 2019), from a model predictive controller that certifies that a safe fallback remains available (Wabersich and Zeilinger, 2021; chapter 15), or from a learned linear model of the constraint, used to project each action onto the safe set (Dalal et al., 2018).
• Safe sets that grow. With a model whose uncertainty is quantified, such as a Gaussian process, the agent explores only where it can be confident of staying within a certified safe region, and enlarges the region as its model improves: for the region of attraction of a controller (Berkenkamp, Turchetta, Schoellig, and Krause, 2017), or for a safety function over the states of a finite MDP (Turchetta, Berkenkamp, and Krause, 2016). Lyapunov functions constructed from a baseline policy’s constraint costs give local constraints under which each policy update keeps the constraint satisfied (Chow, Nachum, Duenez-Guzman, and Ghavamzadeh, 2018).
• Returning and recovering. Many dangers are really irreversibilities. Moldovan and Abbeel (2012) defined safety as the ability to return to the starting states and showed that exploring under this requirement exactly is NP-hard; practical methods learn a reset policy alongside the task policy and abort when the reset would fail (Eysenbach, Gu, Ibarz, and Levine, 2018), or learn a recovery policy that takes over near constraint violations, estimated from offline data (Thananjeyan et al., 2021).
• Learning with the exploration in mind. On-policy learners account for their own exploratory mistakes, which is why SARSA takes the safer path along the cliff in chapter 7; offline RL (chapter 26) and simulation (below) move the dangerous part of learning away from the real system altogether.

Robustness and risk

Constraints bound expected costs. Two other formulations protect against what the expectation hides. Robust MDPs replace the transition function by a set of possible ones and optimize the worst case over the set; when the uncertainty is independent across states, a condition called rectangularity, robust dynamic programming is as tractable as the ordinary kind (Iyengar, 2005; Nilim and El Ghaoui, 2005). Deep RL approximates the worst case by training against an adversary that applies disturbances, as in robust adversarial RL (Pinto, Davidson, Sukthankar, and Gupta, 2017). Worst-case policies can be very conservative, and the right size of the uncertainty set is itself a modeling choice. Risk-sensitive objectives act on the distribution of the return rather than on the model: the conditional value at risk of chapter 17, the expected return in the worst fraction of outcomes, is the most common, and it has a dual interpretation as robustness, since the CVaR equals the worst expected return over a set of reweightings of the outcomes. Distributional critics appear in several of the deployed systems below.

From simulation to reality

The reality gap

Real systems are slow, fragile, and expensive to reset, and most of the agents that act in the physical world today were trained in simulation. Simulators are wrong in ways that matter: actuators have dynamics, friction, and backlash that are hard to model; contacts are approximated; sensors are noisy and delayed; rendered images differ from camera images; and every computation and communication adds latency. A policy optimized against a simulator exploits its errors, for the same reason that a policy exploits a learned model (chapter 23) or a proxy reward: optimization goes wherever the objective is highest, including into the gaps between the model and the world. The difference between a policy’s performance in simulation and in reality is the reality gap.

The first defense is to make the simulator better. System identification fits its parameters to data from the real system, as in Lab 7; learned components replace the hardest parts of the physics, such as the actuator network trained on measurements of the real motors of the ANYmal quadruped and placed inside its simulator (Hwangbo et al., 2019); and details such as actuator dynamics and control latency can decide whether a policy transfers at all (Tan et al., 2018). Simulation has also become fast enough to change what is practical: with thousands of robots simulated in parallel on one GPU, a quadruped learned to walk on flat ground in under four minutes and on rough terrain in twenty (Rudin, Hoeller, Reist, and Hutter, 2021).

Randomization and adaptation

The second defense accepts that the simulator is wrong and trains a policy that works across the ways it might be wrong. Domain randomization samples the simulator’s uncertain properties anew in every episode. Randomizing textures, lighting, and camera positions, Tobin et al. (2017) trained an object detector on non-realistic rendered images alone that located objects on a real table to within 1.5 cm, accurate enough for grasping, and a drone learned to fly through real buildings from randomized renderings without a single real image (Sadeghi and Levine, 2017). Randomizing 95 dynamical parameters, from link masses and joint damping to the friction of the puck and the length of the time step, Peng, Andrychowicz, Zaremba, and Abbeel (2018) trained a recurrent policy to push a puck with a real robot arm in 89% of trials, where a policy trained without randomization never succeeded. OpenAI’s robot hand (OpenAI et al., 2020) learned dexterous in-hand rotation of a block, and later solved a Rubik’s cube (OpenAI et al., 2019), 60% of the time for scrambles needing 15 face rotations and 20% for the hardest, 26-rotation scrambles, with automatic domain randomization, which widens the range of each parameter whenever the policy performs well at its edge, a curriculum over simulators.

Randomization changes the problem the policy solves. The true system becomes a hidden variable drawn from the randomization distribution, which makes the problem partially observable (chapter 14), and the optimal policy is Bayes-adaptive in the sense of chapter 29: it should infer the system from its history and act accordingly. A memoryless policy cannot, and must instead be robust, acceptable on every system in the range, which usually means cautious. A policy with memory can identify the system implicitly, which is why Peng et al.’s recurrent policy transferred where feedforward ones did less well. Rapid motor adaptation (RMA; Kumar, Fu, Pathak, and Malik, 2021) makes the identification explicit. A base policy is trained in simulation with access to a vector of extrinsics, an encoding of the true friction, payload, terrain, and motor strength, and an adaptation module is then trained, also in simulation, to estimate the extrinsics from the last half second of states and actions; deployed on a real quadruped without any fine-tuning, the pair adapted within fractions of a second to terrains and loads it had never seen. The same teacher–student structure, a teacher with privileged information distilled into a student that must infer it, also carried quadruped locomotion over challenging natural terrain (Lee et al., 2020; Miki et al., 2022), and a transformer over the history of observations and actions, adapting in context, walked a full-sized humanoid outdoors after training only in simulation (Radosavovic et al., 2024). The next code compares the three approaches on a problem small enough to test exhaustively: balancing a pendulum whose actuator strength and control delay in “reality” differ from the simulator’s.

import numpy as np

# Simulation to reality with a pendulum balanced upright. theta'' = 10 sin(theta) + u(t - delay) / m, time step
# 0.02 s, torque |u| <= 5, small process noise, 200 steps from |theta| <= 0.2. The cost per step is
# theta^2 + 0.1 theta'^2 + 0.0001 u^2, and falling past 1 radian costs 1 per remaining step. Policies are linear,
# u = -k1 theta - k2 theta', and are trained by searching a grid of gains in simulation:
#   nominal:     in the nominal simulator, m = 1 and no delay;
#   randomized:  on simulators with m uniform on [0.5, 2] and a delay of 0 to 3 steps (0 to 60 ms);
#   adaptive:    a policy that knows the parameters (the best gains for each of a grid of 7 x 4 simulators) combined
#                with online identification: it keeps the likelihood of each simulator given the transitions seen so
#                far and acts with the gains of the most likely one (after 5 steps with the randomized gains);
#   oracle:      the best gains for the true parameters, a reference no deployable policy can know.
# Each is then tested on "real" systems with other parameters, with the same 400 initial states and noise.
DT, T, SIG = 0.02, 200, 0.02
k1, k2 = np.meshgrid(np.linspace(10, 400, 40), np.linspace(1, 60, 30))
GAINS = np.stack([k1.ravel(), k2.ravel()], 1)                       # 1,200 candidate policies


def rollout(m, d, n, seed, gains=None, adapt=None):
    """Mean cost over n episodes on systems (m[i], d[i]); fixed gains (P, 2) or the adaptive policy (P = 1)."""
    rng = np.random.default_rng(seed)
    P = 1 if gains is None else len(gains)
    th, om = np.tile(rng.uniform(-0.2, 0.2, n), (P, 1)), np.zeros((P, n))
    past_u, cost, alive = np.zeros((P, n, 8)), np.zeros((P, n)), np.ones((P, n), bool)
    loglik = None if adapt is None else np.zeros((n, len(adapt[0])))
    for t in range(T):
        if adapt is None:
            K = gains[:, None, :]
        else:
            cand_m, cand_d, cand_K, K_start = adapt
            K = (K_start[None] if t < 5 else cand_K[loglik.argmax(1)])[None]
        u = np.clip(-K[..., 0] * th - K[..., 1] * om, -5, 5)
        past_u = np.roll(past_u, 1, axis=2); past_u[..., 0] = u
        u_applied = np.take_along_axis(past_u, np.broadcast_to(d, (P, n))[..., None], 2)[..., 0]
        om_next = om + DT * (10 * np.sin(th) + u_applied / m) + SIG * np.sqrt(DT) * rng.normal(size=n)
        if adapt is not None:                                   # Gaussian likelihood of the observed transition
            pred = om[0][:, None] + DT * (10 * np.sin(th[0])[:, None] + past_u[0][:, cand_d] / cand_m)
            loglik -= (om_next[0][:, None] - pred) ** 2
        th, om = th + DT * om, om_next
        fell = alive & (np.abs(th) > 1)
        cost += np.where(fell, T - t, np.where(alive, th ** 2 + 0.1 * om ** 2 + 1e-4 * u ** 2, 0)); alive &= ~fell
    return cost.mean(1)


def best_gains(m, d, n=64, seed=0):
    return GAINS[rollout(np.broadcast_to(m, n), np.broadcast_to(d, n), n, seed, GAINS).argmin()]


rng = np.random.default_rng(1)
K_nom = best_gains(1.0, 0)
K_rand = GAINS[rollout(rng.uniform(0.5, 2, 256), rng.integers(0, 4, 256), 256, 1, GAINS).argmin()]
cm, cd = np.meshgrid(np.geomspace(0.5, 2, 7), np.arange(4)); cm, cd = cm.ravel(), cd.ravel()
adapt = (cm, cd, np.array([best_gains(m, d) for m, d in zip(cm, cd)]), K_rand)
print(f"gains (k1, k2): nominal {K_nom.round(1)}, randomized {K_rand.round(1)}")
print("  real system         nominal   randomized   adaptive     oracle     (mean cost per episode, 400 episodes)")
for m, d in ((1, 0), (1, 2), (0.5, 0), (0.5, 2), (2, 3), (1, 5), (0.5, 4)):
    test = lambda **kw: rollout(np.full(400, m), np.full(400, d), 400, 7, **kw)[0]
    row = [test(gains=K_nom[None]), test(gains=K_rand[None]), test(adapt=adapt), test(gains=best_gains(m, d)[None])]
    print(f"  m = {m:3}, {20 * d:3d} ms" + "".join(f"{x:11.2f}" for x in row))
# gains (k1, k2): nominal [90.  29.5], randomized [40.   9.1]
#   real system         nominal   randomized   adaptive     oracle     (mean cost per episode, 400 episodes)
#   m =   1,   0 ms       0.25       0.27       0.26       0.25
#   m =   1,  40 ms       0.50       0.33       0.30       0.30
#   m = 0.5,   0 ms       0.23       0.25       0.23       0.23
#   m = 0.5,  40 ms       2.40       0.34       0.31       0.28
#   m =   2,  60 ms       0.54       0.57       0.54       0.53
#   m =   1, 100 ms       3.30       0.82       2.10       0.47
#   m = 0.5,  80 ms       7.51       6.54       0.86       0.42

Trained on the nominal simulator, the policy uses high gains, which are optimal there: it balances the nominal system as well as the oracle. With a 40 ms delay its cost doubles, and if the actuator is also twice as strong as modeled, the effective gain doubles as well and the cost is more than eight times the oracle’s, the classic way a controller tuned in simulation oscillates on hardware. The randomized policy uses much lower gains, costs slightly more on the nominal system, and stays close to the oracle on the other systems of the table within its training range, though the figure below shows it slipping at the range’s hardest corner, a strong actuator with a 60 ms delay: robustness, bought with a little caution. The adaptive policy identifies the system within a fraction of a second and then acts with gains tuned for it, and comes within about 10% of the oracle on the systems of the table within the training range, and within about 20% at the range’s hardest corner in the figure. Outside it, the two fail differently. With an 80 ms delay and a strong actuator, the adaptive policy recognizes the strong actuator and the longest delay it knows, 60 ms, and its gentle gains for that case still work, at twice the oracle’s cost, while the randomized policy’s compromise, too aggressive for this system, costs fifteen times the oracle’s. With a 100 ms delay and the nominal actuator, the adaptive policy’s best explanation is a 60 ms delay with a weaker actuator, whose gains are too aggressive for the real delay, and it does worse than the randomized policy. An adaptive policy is only as good as the family of systems it can recognize, and outside that family, whether it degrades more or less gracefully than a robust policy depends on what its misidentification does to the gains (exercise 31.7).

Left: the hazard cost of the current policy during Lagrangian learning on the constrained gridworld of the third code. With gradient ascent on the multiplier (orange), the policy never settles, swinging toward one and then the other of two deterministic policies, one above and one below the budget, with its cost cycling between about 0.24 and 0.33; with the PI controller (blue), it settles on the randomized policy that spends the budget exactly. Right: the pendulum of the fourth code with an actuator twice as strong as the simulator’s, against the control delay. The nominal policy degrades as soon as there is any delay; the randomized policy holds up to 40 ms, slips at the edge of its training range (shaded), and fails beyond it; the adaptive policy stays near the oracle through 60 ms and degrades beyond it, but more slowly than the others.

Lab 17 repeats these experiments with deep networks: PPO under a hazard budget with fixed penalties and learned multipliers, a model-based shield compared with learning to avoid the hazard, and a swing-up pendulum trained with domain randomization, memory, and RMA and tested on systems with other actuators and delays.

Learning on real hardware

Some systems cannot be simulated well enough, and some tasks are easier to learn directly than to model. Learning on the real system became practical as algorithms became more sample-efficient. QT-Opt (Kalashnikov et al., 2018) learned vision-based grasping with a distributed Q-learning method from over 580,000 grasp attempts on seven robots, about 800 robot hours over four months, and grasped unseen objects 96% of the time. Soft actor-critic learned a quadruped gait from scratch on a real Minitaur robot in about two hours (Haarnoja et al., 2019); with the high update-to-data ratios of chapter 21, a quadruped learned to walk outdoors in 20 minutes (Smith, Kostrikov, and Levine, 2023), and with a world model, a quadruped learned to roll over, stand, and walk in one hour without resets (Wu, Escontrela, Hafner, Goldberg, and Abbeel, 2022). For manipulation, SERL (Luo et al., 2024) packaged the ingredients, off-policy learning seeded with a few demonstrations, rewards from a learned success classifier, and a forward–backward controller pair that resets the task, and learned circuit-board insertion, cable routing, and object relocation in 25 to 50 minutes on average; with occasional human corrections during training, HIL-SERL (Luo, Xu, Wu, and Levine, 2025) reached near-perfect success on precise and dexterous tasks such as inserting a RAM stick or assembling a car dashboard, most of them within one to two and a half hours of training. The same pattern, a pretrained policy improved by RL on real experience with human interventions, drives the RL fine-tuning of robot foundation models in chapter 29.

Real-world learning adds engineering problems that simulation hides. Episodes must be reset, by a person, by a scripted routine, or by a learned reset policy. Rewards must be measured, often by a learned classifier on camera images, which can itself be exploited. Actions must be smooth, since jittery policies that are harmless in simulation waste power and wear out motors; penalizing the change between successive actions, and between the actions for nearby states, cut the power consumption of a real quadrotor’s learned controller by almost 80% (Mysore, Mabsout, Mancuso, and Saenko, 2021). And someone must be ready to stop the system, which is why the safety filters of the previous section are standard equipment.

Deployed systems

Case studies

Reinforcement learning now controls, or has controlled, a range of real systems. The table collects the best-documented cases; the largest deployment of all, the training of language models, is the subject of chapter 28.

Some entries deserve a caution. The chip-floorplanning results were challenged by an independent assessment, which found that the method did not beat strong academic and commercial placers on the open benchmarks it built (Cheng, Kahng, Kundu, Wang, and Wang, 2023), a finding a later meta-analysis endorsed (Markov, 2024); the authors replied that the replications omitted pretraining and used far less compute (Goldie, Mirhoseini, and Dean, 2024), and the journal appended an addendum to the paper in 2024. The episode is a reminder that “RL beats the experts” claims on real problems depend heavily on baselines, benchmarks, and compute, as they do in simulation (chapter 21).

What the successes share

The deployed systems have more in common than their algorithms. Nearly all had a simulator that was either very good (the game, the compiler, the codec) or engineered for the purpose with real data, and nearly all trained most or all of their experience there. Their rewards measured something the operators cared about directly and could not easily be gamed: time near the station, lap time, energy, bitrate at fixed quality, latency of verified-correct code. Their learned components were embedded in larger engineered systems, with conventional controllers, safety checks, and human operators around them; Google’s data-center cooling system, for example, vetted every action against operator-defined constraints, had the local control system check it again, discarded actions it was not confident about, and let operators leave AI control at any time (Gamble and Gao, 2018). And they were evaluated extensively before deployment, in simulation or on logged data with the off-policy evaluation methods of chapter 26. Several also used distributional critics, and several used asymmetric designs, in which training uses information that deployment will not have, the privileged critic of the tokamak controller and the teacher of RMA.

The challenges of real-world RL

Dulac-Arnold, Mankowitz, and Hester (2019) listed nine challenges that separate real-world problems from benchmarks, and studied them empirically in a suite of perturbed control tasks (Dulac-Arnold et al., 2021). Most have appeared in this module:

1. Learning offline from fixed logs of a behavior policy: chapter 26.
2. Learning on the real system from limited samples: models (chapter 23) and sample-efficient critics (chapter 21).
3. High-dimensional continuous states and actions: the deep RL of chapters 16 to 21.
4. Safety constraints that should never, or rarely, be violated: this chapter.
5. Partial observability and nonstationarity: chapter 14 and the adaptation of this chapter.
6. Rewards that are unspecified, multi-objective, or risk-sensitive: this chapter and chapter 17.
7. Operators who need explainable policies and actions.
8. Inference in real time at the control frequency of the system.
9. Large or unknown delays in actuators, sensors, or rewards.

The last three need a word. Explainability is often obtained by structure rather than by explanation: a learned policy that proposes setpoints to a conventional controller, or chooses among a few interpretable modes, is easier to trust than one that commands actuators directly. Real-time inference constrains the policy’s size, which is why the tokamak’s actor was small, and why large models are often distilled into small ones for deployment. And time itself is a design parameter. The discount per step and the control frequency together set the horizon in seconds, \(\Delta t/(1-\gamma)\), so a discount tuned at one frequency is wrong at another, and, as chapter 15 noted, discounting can make instability look cheap. As the time step shrinks, the effect of any single action on the return shrinks with it, the action values of all actions converge to the state value, and Q-learning’s choice among them drowns in noise (Tallec, Blier, and Ollivier, 2019; exercise 31.8); learning the advantage rescaled by the time step, rather than the action value, removes the problem. Delays make the observed state non-Markov, since the actions already sent but not yet applied also determine the future; augmenting the state with the last few actions restores the Markov property (Katsikopoulos and Engelbrecht, 2003), and a policy that must act while the world keeps moving can be designed for it from the start (Ramstedt and Pal, 2019).

The two halves of this chapter meet here. Everything that makes the real world hard, misspecified rewards, unmodeled dynamics, constraints that bind during learning, exploits the same property of reinforcement learning: it optimizes exactly what it is given, as well as it can. That property is what made the deployed systems possible, and it is also why the problem of specifying what we want, and of verifying that an optimizer has found it rather than something that merely scores well, becomes more pressing as the optimizers become more capable. It is the central problem of the Safety and Frontier module.

Exercises

Exercise 31.1 — Why the progress bonus hovers

In the gridworld of the first code, \(\gamma=0.98\), entering the goal earns 1, and the progress bonus adds 0.1 to every move that reduces the distance to the goal. (a) From a state next to the goal, compare entering the goal with stepping away and back forever. (b) For which discounts is entering the goal from that state better? (c) Show that with a potential-based term instead, the discounted shaping reward collected around any cycle that leaves \(s\) and returns to it after \(k\) steps is \(-(1-\gamma^k)\Phi(s)\), and explain why cycles then never become optimal.

Exercise 31.2 — Shaping as initialization

(a) Show that Q-learning with potential-based shaping, started from \(Q'_0=0\), and Q-learning without shaping, started from \(Q_0(s,a)=\Phi(s)\), maintain \(Q_t(s,a)=Q'_t(s,a)+\Phi(s)\) and choose the same actions, given the same random numbers. (b) Use this to explain why \(\Phi=-d/28\) slowed learning in the first code while \(\Phi=1-d/28\) sped it up. Consider a move into a wall, which leaves the state unchanged. (c) Show that with \(\Phi=V^*\) the expected shaped reward of each action is its advantage \(A^*(s,a)\), and conclude that a myopic agent, one with \(\gamma=0\), can act optimally.

Exercise 31.3 — The optimism of hindsight

From a start state, an agent can walk, reaching the goal \(g\) in two deterministic steps, or jump, reaching \(g\) in one step with probability \(p\) and otherwise falling into a pit that ends the episode. The reward is 1 on reaching \(g\) and 0 otherwise, with \(\gamma=0.9\). Transitions are relabeled with the “future” strategy with \(k=4\) achieved goals each; after a jump the only achieved state is where it landed. (a) What are the true values of walking and jumping for goal \(g\)? (b) Among the stored transitions of the jump with goal \(g\), what fraction are successes, and what estimate of the jump’s value does Q-learning converge to? (c) For which \(p\) does the agent prefer to jump although walking is better? (d) Why is the walk’s value estimated without bias?

Exercise 31.4 — Why constrained optima randomize

A one-step decision has two actions: fast, with reward 1 and cost 1, and slow, with reward 0 and cost 0. The budget on the expected cost is \(d=0.3\). (a) Find the best deterministic policy that satisfies the budget and the best randomized one. (b) Compute the dual function \(g(\lambda)=\max_\pi L(\pi,\lambda)\) and its minimizer, and verify that there is no duality gap. (c) At the optimal multiplier, which policies maximize the Lagrangian, and which of them is optimal for the constrained problem? What does gradient ascent on \(\lambda\), with best responses to the current multiplier, do?

Exercise 31.5 — The multiplier as a feedback loop

Model the policy’s response to the multiplier as a lag: with \(x_k=J_c(\pi_k)-d\) and \(y_k=\lambda_k-\lambda^*\), suppose \(x_{k+1}=x_k+\beta(-\kappa y_k-x_k)\), where \(-\kappa y\) is the cost violation the policy would settle at for a fixed multiplier, \(\kappa>0\), and \(0<\beta\le1\) is how fast the policy adapts. (a) With the gradient-ascent update \(y_{k+1}=y_k+\eta x_k\), find when the joint dynamics oscillate and when they are stable. (b) With a PI update, \(y_k=K_Px_k+I_k\) and \(I_{k+1}=I_k+\eta x_k\), show that a large enough proportional gain removes the oscillation.

Exercise 31.6 — Options shorten the horizon

A corridor has states \(0,1,\dots,N\), starting at 0, with the goal at \(N\); entering it earns 1 and ends the episode; the primitive actions move left and right, and moving left at 0 stays put. An option “run right” moves right until it reaches \(N\). (a) What is the option’s value from state \(i\) in the semi-MDP, and how does it compare with the optimal primitive value? (b) How many sweeps of synchronous value iteration propagate the goal’s value to state 0 with primitive actions alone, and with the option added? (c) An agent explores by choosing uniformly at random among its choices. Compare the expected number of steps to first reach the goal from 0 with primitive actions alone, \(N(N+1)\) for a walk that stays put with probability 1/2 at 0, with the expected number of decisions when the option is one of three choices. (d) What goes wrong if the agent may use only the options “run right to the end” and “run left to the end”?

Exercise 31.7 — Robust versus adaptive, in one dimension

A system \(x_{t+1}=x_t+bu_t\) has an unknown gain \(b\in[0.5,2]\), and a linear policy \(u_t=-kx_t\) makes \(x_{t+1}=(1-bk)x_t\). The cost is \(\sum_{t\ge0}x_t^2\) from a given \(x_0\ne0\). (a) What does the policy that is optimal for the nominal \(b=1\) cost on the systems \(b=0.5\) and \(b=2\)? (b) Find the gain that minimizes the worst case over the range, and its worst-case cost. What does it cost on the nominal system? (c) An adaptive policy uses the robust gain for one step, identifies \(b\) from the observed change, and then acts optimally. What is its worst-case cost? (d) A real system has \(b=3\), outside the range, and the adaptive policy must choose its estimate within \([0.5,2]\). What do the robust and the adaptive policies do?

Exercise 31.8 — Time steps, discounts, and delays

(a) A controller runs at 50 Hz with \(\gamma=0.99\). What is its horizon in seconds, and what discount keeps the same horizon at 500 Hz? (b) For a continuous-time system \(\dot x=f(x,u)\) with reward rate \(r(x,u)\) and discount rate \(\rho\), a step of length \(\Delta t\) holds the action fixed, earns \(r\Delta t\), and discounts by \(e^{-\rho\Delta t}\). Show that \(Q_{\Delta t}(x,u)=V(x)+\Delta t\,A(x,u)+o(\Delta t)\) for a function \(A\) that does not depend on \(\Delta t\), and explain why Q-learning degrades as \(\Delta t\to0\). (c) Actions take effect \(d\) steps after they are chosen: \(x_{t+1}=F(x_t,a_{t-d})\). Show that \(x_t\) is not a Markov state but \((x_t,a_{t-d},\dots,a_{t-1})\) is.

Appendices