23. Model-Based RL and World Models

Learning models of the world

What a model buys

Every deep agent of the previous chapters is model-free: it learns values or policies directly from experience, and it needs a great deal of it, hundreds of thousands of steps for a simulated cheetah and millions for an Atari game. A model of the environment, a predictor of the next state and reward given the current state and action, changes the economics. Once learned, it can be queried without acting, so each real transition can be reused through planning (chapter 10); it can be optimized through, as optimal control does (chapter 15); it transfers to new rewards in the same environment, since the dynamics do not change with the task; and it is trained by supervised learning, whose data are every transition the agent has ever seen, whatever policy produced it. Lab 7 showed the payoff in a small setting: a model fitted to 1,000 random transitions controlled a pendulum as well as its true dynamics. This chapter develops model-based RL with neural networks: how to learn models whose errors do not destroy the policies built on them, how to use them for planning and for generating experience, how to learn compact latent models from images, and what a model should predict in the first place. It ends with the generative world models that simulate interactive environments from video, and with agents trained entirely inside them. Chapter 24 continues with the search-based planning of AlphaZero and MuZero.

Why errors compound

A model trained to predict one step ahead is used many steps ahead, and its errors compound: each prediction starts from the previous one’s mistake, in a state that the model may never have seen. Worse, an optimizer that plans or learns inside the model seeks out states where the model’s predictions are too good, and those are where the model is most likely wrong, the same exploitation of errors as the actor’s exploitation of its critic in chapter 21. The next code trains an ensemble of five networks on 2,000 random transitions of the pendulum, all from states far from the upright position, and measures the error of open-loop predictions over 30 steps from states like the training data and from states near the top.

import numpy as np
import torch

# Learned dynamics models and why their errors compound. Gymnasium's pendulum (theta'' = 3g/(2l) sin(theta) + 3u/(ml^2),
# dt = 0.05, speed clipped to [-8, 8]), state (cos, sin, speed). An ensemble of 5 networks, each trained on its own
# bootstrap sample of 2,000 transitions from random torques starting near the bottom (|theta| > 2), predicts the
# change of the state. Then: the error of open-loop rollouts of 1 to 30 steps from states like the training data and
# from states near the top (never seen), and the ensemble's disagreement as a signal of that error.
torch.set_num_threads(1); torch.manual_seed(0); rng = np.random.default_rng(0)
g, m, l, dt = 10.0, 1.0, 1.0, 0.05


def step(th, thd, u):
    thd = np.clip(thd + (3 * g / (2 * l) * np.sin(th) + 3.0 / (m * l ** 2) * u) * dt, -8, 8)
    return th + thd * dt, thd


def obs(th, thd):
    return np.stack([np.cos(th), np.sin(th), thd / 8], -1)


def data(n, near_top):
    th = rng.uniform(-0.5, 0.5, n) if near_top else rng.choice([-1, 1], n) * rng.uniform(2.0, np.pi, n)
    thd, u = rng.uniform(-2, 2, n), rng.uniform(-2, 2, n)
    th2, thd2 = step(th, thd, u)
    return obs(th, thd), u, obs(th2, thd2)


X, U, X2 = data(2000, near_top=False)
inp = torch.as_tensor(np.column_stack([X, U / 2]), dtype=torch.float32)
out = torch.as_tensor(X2 - X, dtype=torch.float32)
ensemble = []
for k in range(5):
    idx = torch.as_tensor(rng.integers(len(X), size=len(X)))
    net = torch.nn.Sequential(torch.nn.Linear(4, 64), torch.nn.Tanh(), torch.nn.Linear(64, 64), torch.nn.Tanh(), torch.nn.Linear(64, 3))
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    for it in range(1500):
        b = idx[torch.randint(len(idx), (256,))]
        loss = ((net(inp[b]) - out[b]) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    ensemble.append(net)


def rollout_errors(near_top, H=30, n=500):
    th = rng.uniform(-0.5, 0.5, n) if near_top else rng.choice([-1, 1], n) * rng.uniform(2.0, np.pi, n)
    thd = rng.uniform(-2, 2, n); u = rng.uniform(-2, 2, (H, n))
    x_true = obs(th, thd); x_pred = [torch.as_tensor(x_true, dtype=torch.float32) for _ in ensemble]
    errs, spread = [], []
    for h in range(H):
        th, thd = step(th, thd, u[h]); x_true = obs(th, thd)
        with torch.no_grad():
            ui = torch.as_tensor(u[h][:, None] / 2, dtype=torch.float32)
            x_pred = [x + net(torch.cat([x, ui], 1)) for x, net in zip(x_pred, ensemble)]
        P = torch.stack(x_pred).numpy()
        errs.append(np.sqrt(((P.mean(0) - x_true) ** 2).sum(1)).mean())
        spread.append(np.sqrt(P.var(0).sum(1)).mean())
    return np.array(errs), np.array(spread)


print("   open-loop prediction error (and ensemble spread) after h steps, mean over 500 random starts and torques")
print("   states              h=1              h=5             h=10             h=30")
for near_top, name in ((False, "like the training"), (True, "near the top")):
    e, s = rollout_errors(near_top)
    print(f"   {name:17s}" + "".join(f"   {e[h - 1]:.3f} ({s[h - 1]:.3f})" for h in (1, 5, 10, 30)))
#    open-loop prediction error (and ensemble spread) after h steps, mean over 500 random starts and torques
#    states              h=1              h=5             h=10             h=30
#    like the training   0.002 (0.002)   0.011 (0.009)   0.030 (0.020)   0.064 (0.035)
#    near the top        0.110 (0.035)   0.531 (0.152)   1.060 (0.276)   1.396 (0.422)

On states like its training data, the model’s one-step error is 0.002, and it compounds to 0.064 after 30 steps, thirty times more. Near the top, a region its data never visited, the one-step error is already fifty times larger, and after ten steps the predictions are essentially unrelated to the truth. The ensemble’s disagreement, the spread of its members’ predictions, grows in the same way: it is small where the model is right and large where it is wrong, which makes it a usable warning (the figure below). Uncertainty comes in two kinds, and a model-based agent must represent both. Aleatoric uncertainty is the environment’s own randomness, which more data cannot reduce; a network that outputs a distribution, such as a Gaussian with a learned variance, captures it. Epistemic uncertainty is the model’s ignorance where data are scarce, which more data would reduce; disagreement among ensemble members trained on different samples or from different initializations captures it, as the ensembles of chapter 22 did for values.

Left: open-loop prediction error of an ensemble of five networks trained on 2,000 random transitions of the pendulum, as a function of the number of steps predicted, for starting states like the training data and near the upright position, which the data never visited (solid), with the ensemble’s disagreement (dashed). Right: for each of 500 starting states, the disagreement and the actual error after ten steps. Disagreement ranks the errors correctly, and separates the two regions, but underestimates the error far from the data.

Planning and learning with learned models

Probabilistic ensembles and planning

The first deep model-based agents to match model-free ones on continuous control combined the two kinds of uncertainty with model predictive control. PETS (Chua, Calandra, McAllister, and Levine, 2018) learns an ensemble of networks, each outputting a Gaussian over the next state, and at every step plans by the cross-entropy method (chapter 15): it samples action sequences, evaluates each by propagating particles through the ensemble, with each particle following a randomly chosen member, and executes the first action of the optimized sequence. Averaging over members penalizes plans that only some members consider good, which keeps the optimizer away from the model’s errors. PETS reached the asymptotic performance of the model-free methods of its time on several MuJoCo tasks with a small fraction of their data. It inherited earlier lessons: PILCO (Deisenroth and Rasmussen, 2011) had used Gaussian-process models with analytic uncertainty propagation to swing up a cart-pole from a few trials, and Nagabandi, Kahn, Fearing, and Levine (2018) had shown that neural-network models with MPC learn locomotion quickly, and can initialize a model-free learner that then surpasses them.

Model-based policy optimization

Planning at every step is expensive, and planning over long horizons compounds errors. Dyna-style methods (chapter 10) instead use the model to generate experience for a model-free learner. MBPO (Janner, Fu, Zhang, and Levine, 2019) found the key to making this work with neural models: short branched rollouts. It starts model rollouts from states in the real replay memory, runs them for only \(k\) steps (from 1 up to 15 or 25 depending on the task, lengthened on a fixed linear schedule as training progresses), and trains a soft actor–critic on a mixture of real and model data, with many updates per real step. Starting from real states keeps the rollouts inside the data distribution, and short rollouts limit the compounding of errors, while the model still multiplies the data available near every state the agent has visited. The authors derived a bound on the policy’s true return in terms of the model’s error and the length of the rollouts that motivates this design (exercise 23.3). MBPO learned the MuJoCo locomotion tasks several times faster than SAC, an order of magnitude faster on some (on Ant it matched SAC’s 3-million-step performance after 300,000 steps), and the model-free REDQ of chapter 21 later matched its efficiency with an ensemble of critics and a high replay ratio, which suggests that a large part of its gain comes from the many updates per real sample that the model makes possible.

A model can also sharpen the value targets rather than add data. Model-based value expansion (Feinberg et al., 2018) computes \(h\)-step targets with the model’s rewards before bootstrapping, and STEVE (Buckman, Hafner, Tucker, Brevdo, and Lee, 2018) weights targets of different horizons by their uncertainty under an ensemble. And a differentiable model gives the policy gradients directly: stochastic value gradients (Heess et al., 2015) backpropagate the value of a trajectory through the model and the policy, the approach that Dreamer scales up.

Latent world models

Models of observations

Images cannot be predicted pixel by pixel over long horizons, and they do not need to be. World models (Ha and Schmidhuber, 2018) compressed each frame with a variational autoencoder (GenAI chapter 3), predicted the next latent code with a recurrent network that outputs a mixture of Gaussians, and trained a tiny linear controller on the latent code and the recurrent state with an evolution strategy. The agent could be trained entirely inside its own “dream” and transferred to the real game. A dream made less random than the model had learned let the controller exploit its flaws and fail in reality, and a dream made slightly more random transferred best. PlaNet (Hafner et al., 2019) made latent planning work for control from pixels. Its recurrent state-space model (RSSM) has a deterministic recurrent state, which carries information reliably over many steps, and a stochastic latent state, which represents what the model cannot predict; it is trained as a sequential variational autoencoder, reconstructing observations and rewards, with a KL term between the posterior over the latent state given the current observation and the prior predicted from the past. PlaNet planned by the cross-entropy method in latent space and solved visual control tasks from the DeepMind Control Suite with far fewer episodes than model-free agents.

Learning behaviors in imagination: the Dreamer agents

Dreamer (Hafner, Lillicrap, Ba, and Norouzi, 2020) replaced planning at every step by an actor and a critic trained entirely on imagined trajectories: starting from latent states of real sequences, it rolls the RSSM forward for 15 steps with the actor’s actions, computes λ-returns from predicted rewards and the critic’s values, and trains the actor by backpropagating these returns through the learned dynamics, a value gradient in the spirit of SVG. DreamerV2 (Hafner, Lillicrap, Norouzi, and Ba, 2021) used discrete latent variables, 32 categorical variables of 32 classes, and KL balancing, which trains the prior faster than the posterior, and was the first agent to reach human-level performance on the Atari benchmark by learning behaviors inside a separately trained world model, on a single GPU. DreamerV3 (Hafner, Pasukonis, Ba, and Lillicrap, 2025) made the recipe robust across domains with one set of hyperparameters: symlog predictions that compress large values, a two-hot categorical representation of rewards and values, returns normalized by their percentiles, and free bits that stop the KL term from collapsing the latent (exercise 23.8). It outperformed specialized methods across more than 150 tasks, from Atari to continuous control to 3D worlds, and it was the first algorithm to collect diamonds in Minecraft from scratch, without human data or curricula. The Dreamer recipe also made learning on real robots practical: with DayDreamer (Wu et al., 2022), which applied DreamerV2 to physical robots, a quadruped learned to walk from scratch in about an hour of real-world experience.

What should a model predict?

Reconstructing observations spends a model’s capacity on everything in them, including what does not matter for control: the texture of a floor, the leaves of a tree, a flickering screen. Maximum-likelihood models also trade errors in ways unrelated to the task: a model can be more accurate on average and worse for control, the objective mismatch of Lambert, Amos, Yadan, and Calandra (2020). The alternative is to train models only to predict what planning needs. A model is value-equivalent to the environment for a set of policies and value functions if applying the model’s Bellman operators to them gives the same results as the environment’s (Grimm, Barreto, Singh, and Silver, 2020; see also Farahmand, Barreto, and Nikovski, 2017): it may be wrong about everything else. The next code makes the difference concrete with a linear system hidden among irrelevant, high-variance distractors, and three latent models of the same small size.

import numpy as np
import scipy.linalg
import torch

# What should a world model predict? A 2-D controllable system, x' = A x + B u (a discretized double integrator),
# with reward -(x_1^2 + 0.1 x_2^2) - 0.01 u^2, observed together with 20 irrelevant, slowly varying distractors of
# much larger variance (they carry about 98% of the observations' variance), mixed by a random rotation into 22
# dimensions. Three latent models with a 2-D state z = W o, linear latent dynamics z' = F z + G u, and a quadratic
# reward model are trained on 20,000 random transitions (random controls, restarts every 50 steps) by rolling the
# latent model 10 steps ahead:
#   reconstruction only: the latent is shaped only by predicting the future observations (through a linear
#     decoder); the reward model is fitted on top of it;
#   reconstruction and reward: both losses shape the latent, as in Dreamer;
#   value-equivalent: only the future rewards, as in MuZero and TD-MPC.
# Each model is then used for control: the LQR of the latent model, applied through z = W o. Report the true
# average cost per step, against the optimal controller that knows the true state and against doing nothing.
torch.set_num_threads(1); torch.manual_seed(0); rng = np.random.default_rng(0)
A = np.array([[1.0, 0.1], [0.0, 1.0]]); B = np.array([[0.005], [0.1]])
Qx, Ru, nd = np.diag([1.0, 0.1]), 0.01, 20
M = np.linalg.qr(rng.normal(size=(2 + nd, 2 + nd)))[0]            # the mixing rotation


def simulate(n):
    x, d = rng.normal(size=2), 5 * rng.normal(size=nd)
    O, U, Rw = [], [], []
    for t in range(n):
        o = M @ np.concatenate([x, d])
        u = rng.normal()
        r = -(x @ Qx @ x) - Ru * u ** 2
        x = A @ x + B[:, 0] * u + 0.01 * rng.normal(size=2); d = 0.98 * d + rng.normal(size=nd)
        if t % 50 == 49:
            x = rng.normal(size=2)
        O.append(o); U.append(u); Rw.append(r)
    return np.array(O), np.array(U), np.array(Rw)


O, U, R = simulate(20000)
T = torch.as_tensor
O_, U_, R_ = T(O, dtype=torch.float32), T(U, dtype=torch.float32), T(R, dtype=torch.float32)


def train(reconstruct, reward_shapes_latent=True, H=10, steps=4000):
    W = torch.nn.Parameter(0.1 * torch.randn(2, 2 + nd)); F = torch.nn.Parameter(torch.eye(2)); G = torch.nn.Parameter(torch.zeros(2, 1))
    L = torch.nn.Parameter(torch.eye(2)); c = torch.nn.Parameter(torch.tensor(0.0)); D = torch.nn.Parameter(0.1 * torch.randn(2 + nd, 2))
    opt = torch.optim.Adam([W, F, G, L, c, D], lr=1e-2)
    for it in range(steps):
        idx = torch.randint(len(O) - H, (256,))
        z, loss = O_[idx] @ W.T, 0.0
        for h in range(H):
            u = U_[idx + h][:, None]
            zr = z if reward_shapes_latent else z.detach()
            r_hat = -((zr @ L) ** 2).sum(1) - c ** 2 * u[:, 0] ** 2         # a negative-definite quadratic reward
            loss = loss + ((r_hat - R_[idx + h]) ** 2).mean()
            if reconstruct:
                loss = loss + ((z @ D.T - O_[idx + h]) ** 2).sum(1).mean()
            z = z @ F.T + u @ G.T
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        Qz = (L @ L.T).numpy().astype(float); Rz = float(c ** 2) + 1e-6
        Fz, Gz = F.numpy().astype(float), G.numpy().astype(float)
    P = scipy.linalg.solve_discrete_are(Fz, Gz, Qz + 1e-6 * np.eye(2), np.array([[Rz]]))
    K = np.linalg.solve(Rz + Gz.T @ P @ Gz, Gz.T @ P @ Fz)
    return K, W.detach().numpy().astype(float)


def cost(K=None, W=None, true_state=False, n=4000):
    x, total = rng.normal(size=2), 0.0
    d = 5 * rng.normal(size=nd)
    for t in range(n):
        o = M @ np.concatenate([x, d])
        u = 0.0 if K is None else (-K @ (x if true_state else W @ o)).item()
        total += x @ Qx @ x + Ru * u ** 2
        if abs(x).max() > 1e6:
            return np.inf                                           # the closed loop is unstable
        x = A @ x + B[:, 0] * u + 0.01 * rng.normal(size=2); d = 0.98 * d + rng.normal(size=nd)
        if t % 200 == 199:
            x = rng.normal(size=2)
    return total / n


P = scipy.linalg.solve_discrete_are(A, B, Qx, np.array([[Ru]]))
K_opt = np.linalg.solve(Ru + B.T @ P @ B, B.T @ P @ A)
print("   controller                                   average cost per step")
print(f"   no control                                   {cost():10.3f}")
print(f"   optimal, with the true state                 {cost(K_opt, true_state=True):10.3f}")
for name, rec, shaped in (("latent model: reconstruction only", True, False),
                          ("latent model: reconstruction and reward", True, True),
                          ("latent model: value-equivalent", False, True)):
    K, W = train(rec, shaped)
    J = cost(K, W)
    print(f"   {name:44s} " + (f"{J:10.3f}" if np.isfinite(J) else "  unstable"))
#    controller                                   average cost per step
#    no control                                      122.504
#    optimal, with the true state                      0.023
#    latent model: reconstruction only              unstable
#    latent model: reconstruction and reward           0.267
#    latent model: value-equivalent                    0.112

The controllable part of the observation carries about 2% of its variance. A latent model shaped only by reconstructing future observations spends its two dimensions on the distractors, which explain most of the variance, and the controller built from it, acting on the wrong quantities through feedback it misunderstands, drives the system unstable. Adding the reward loss to the reconstruction pulls the latent partway toward the relevant state, and the controller works, at more than ten times the optimal cost. The value-equivalent model, trained only to predict future rewards under the actions taken, ignores the distractors and controls best of the three, at about five times the optimal cost; with its two dimensions it cannot represent the state perfectly either, but it spends them on what matters. The same logic underlies MuZero (chapter 24) and the TD-MPC agents.

TD-MPC

TD-MPC (Hansen, Wang, and Su, 2022) combines a value-equivalent latent model with planning. Its model encodes the observation into a latent state and is trained, over short multi-step rollouts in latent space, with three losses: predicting the rewards, predicting TD targets for a learned action-value function, and a latent consistency loss that makes the predicted latent states match the encodings of the observations actually reached, with no reconstruction at all. It acts by MPPI planning over a short horizon in latent space, with the learned value function as the terminal value beyond it, and a learned policy to propose action sequences. TD-MPC2 (Hansen, Su, and Wang, 2024) scaled the recipe with normalization of the latent state (SimNorm), discrete regression of values and rewards, and a single set of hyperparameters, to 104 continuous-control tasks, and trained one agent with 317 million parameters on 80 tasks across several domains and embodiments.

Generative world models

Transformers and diffusion as world models

The generative models of the previous module have become world models. IRIS (Micheli, Alonso, and Fleuret, 2023) tokenizes each frame with a discrete autoencoder and predicts the next tokens, reward, and termination with an autoregressive transformer (GenAI chapter 2), and trains its policy entirely in imagination; it exceeded human performance on Atari 100k by the mean human-normalized score. DIAMOND (Alonso et al., 2024) generates the next frame with a diffusion model (GenAI chapter 7), argues that visual details lost by discrete latents matter for control, and reached a mean human-normalized score of 1.46 on Atari 100k, the best for an agent trained entirely inside a world model at the time; trained on recorded play, it also ran as an interactive, playable model of a 3D game. GameNGen (Valevski, Leviathan, Arar, and Fruchter, 2025) simulated the game DOOM with a diffusion model in real time, at 20 frames per second on a single TPU.

Generative interactive environments

Genie (Bruce et al., 2024) learned an interactive environment from unlabeled Internet videos of 2D platform games: a latent action model infers a small discrete set of actions (eight) from the past frames and the next frame, and a spatiotemporal transformer predicts the next frame’s tokens from the past frames and the latent action, so that a user, or an agent, can step through a world generated from a single image. Its successors extended the idea to 3D worlds that stay consistent for longer, and Genie 3 (DeepMind, 2025) generates navigable environments from text prompts in real time, at 24 frames per second and 720p, consistent for a few minutes. Such models are a new kind of environment for training agents: unlimited, controllable, and generated on demand, though only as faithful to real physics as their training video. Dreamer 4 (Hafner, Yan, and Lillicrap, 2025) closed the loop: a scalable transformer world model, trained with a shortcut forcing objective that lets it generate in a few steps, runs in real time on a single GPU, and an agent trained purely inside it, from offline data without any interaction with the game, obtained diamonds in Minecraft.

When models help

Models help most where data are expensive and the dynamics are learnable: robots, slow simulators, benchmarks that limit interaction. They cost computation, since the model must be trained and queried, and they add their own failure modes: compounding errors, exploitation by the planner or the policy, and capacity spent on irrelevant detail. For a long time, model-based agents learned fast but plateaued below model-free ones; the methods of this chapter have largely closed that gap, and in data-limited regimes they lead. Moerland, Broekens, Plaat, and Jonker (2023) survey the design space. Lab 12 builds probabilistic ensembles, plans with them, and uses them to generate experience for a model-free learner.

Exercises

Exercise 23.1 — How errors compound

A linear system \(x_{t+1}=Ax_t\) is modeled by \(\hat A=A+E\) with \(\|E\|\le\epsilon\). (a) Show that the error of an \(h\)-step open-loop prediction satisfies \(\|\hat A^hx_0-A^hx_0\|\le\epsilon\sum_{k=0}^{h-1}\|\hat A\|^k\|A\|^{h-1-k}\|x_0\|\). (b) Evaluate the bound for \(\|A\|=\|\hat A\|=\rho\) with \(\rho<1\), \(\rho=1\), and \(\rho>1\), and relate it to the code’s two regions.

Exercise 23.2 — The simulation lemma

Let \(M\) and \(\hat M\) be two MDPs with the same states and actions, rewards in \([0,R_{\max}]\), and \(\|P(\cdot\mid s,a)-\hat P(\cdot\mid s,a)\|_1\le\epsilon_P\) and \(|r(s,a)-\hat r(s,a)|\le\epsilon_R\) everywhere. Show that for every policy, \(\|V^\pi_M-V^\pi_{\hat M}\|_\infty\le\frac{\epsilon_R}{1-\gamma}+\frac{\gamma\epsilon_PR_{\max}}{(1-\gamma)^2}\).

Exercise 23.3 — Why short branched rollouts

MBPO’s analysis bounds the true return of the policy by its return under branched model rollouts minus two penalties: one proportional to the rollout length \(k\) times the model’s error, and one proportional to the divergence between the current policy and the policy that collected the data, discounted by \(\gamma^k\), so that it shrinks as \(k\) grows. (a) Why does starting rollouts from states in the replay memory, rather than from the initial state, change the trade-off? (b) Why does MBPO lengthen the rollouts as training progresses?

Exercise 23.4 — Two kinds of uncertainty

A robot pushes a puck on a table whose friction varies randomly from push to push. (a) Which part of the uncertainty in the puck’s next position is aleatoric and which epistemic? (b) What would a single network that outputs a Gaussian report in a region of states it has never seen, and what does an ensemble of such networks add? (c) PETS’s TS∞ variant propagates each particle through one randomly chosen member for the whole rollout. What does this gain over averaging the members at every step?

Exercise 23.5 — Imagination and value gradients

Dreamer trains its actor by backpropagating the λ-return of imagined trajectories through the learned dynamics. (a) Compare this gradient with the score-function (REINFORCE) gradient of chapter 13 in terms of variance and bias. (b) What happens if the model’s dynamics are wrong in a way that favors some action?

Exercise 23.6 — Multi-step losses

(a) Why do MuZero and TD-MPC train their latent models by rolling them several steps forward from an encoded state, rather than only one step? (b) What does TD-MPC’s latent consistency loss prevent that a reward loss alone would not?

Exercise 23.7 — Reading the value-equivalence experiment

In the code on what a model should predict: (a) why does the reconstruction-only model make the system unstable rather than merely fail to control it? (b) The value-equivalent model does not reach the optimal cost. Give two reasons.

Exercise 23.8 — Symlog and two-hot

DreamerV3 predicts quantities with \(\operatorname{symlog}(x)=\operatorname{sign}(x)\ln(|x|+1)\) and represents rewards and values as two-hot vectors over bins \(b_1<\dots<b_K\): a scalar \(y\) between \(b_k\) and \(b_{k+1}\) puts weight \((b_{k+1}-y)/(b_{k+1}-b_k)\) on bin \(k\) and the rest on bin \(k+1\). (a) Give the inverse of symlog, and compare it with the value rescaling of R2D2 (exercise 18.6). (b) Show that the two-hot vector’s expectation over the bins is \(y\), and explain why training a categorical output with cross-entropy against it is robust to the scale of the targets.

Appendices