Figure sources

Every figure in the Generative AI chapters is an original plot generated for these notes on 24 September 2026 with NumPy, SciPy, Matplotlib, PyTorch, and scikit-learn; no external artwork or course slides are reproduced. Each chapter’s figures come from one script in Sources/Figure code, which is the complete record of the construction. The shared module genfig.py sets the palette, fonts, and export (white background, 200 dpi PNG). The scripts of chapters 1–3, 7, and 10–12 use scikit-learn’s bundled \(8\times8\) digits (1,797 images with intensities 0–16) with the module’s split: a permutation by NumPy default_rng(0), the first 1,500 images for training and the other 297 held out. Chapters 2, 3, and 12 binarize the digits at intensity \(>7\), and chapters 7, 10, and 11 scale them to \([-1,1]\) by \(x/8-1\). The other figures use synthetic distributions generated in the scripts: two moons, mixtures of Gaussians on a line and on a circle, points on a circle, and demonstrations of a steering task. No script loads a file or depends on another, and every model is trained from scratch when its script runs. Data sources describes the data. The tables below summarize the data, seeds, and model settings of each figure so that it can be read and regenerated without opening the script; the computing setup gives the environment and commands.

Seeds refer to torch.manual_seed (“torch seed”), torch.Generator().manual_seed (“generator seed”), or numpy.random.default_rng (“NumPy seed”); scikit-learn’s random_state is named where it is used, and layers whose initialization is not described keep PyTorch’s default. Draws without their own generator, such as minibatches, training noise, and fresh samples of synthetic data, continue the stream of the script’s torch or NumPy seed in the order of the script. The receptive fields of chapter 2 and the Dirac GAN of chapter 5 involve no randomness. No script reads command-line arguments; each always writes all its figures. The scripts that train with PyTorch set torch.set_num_threads(1), except ch02_autoregressive.py, ch03_vae.py, and ch04_flows.py, which use PyTorch’s default; ch05_gans.py and ch08_sde.py set it before their second figure, the first to use PyTorch. According to their docstrings, ch01_foundations.py and ch13_eval.py take seconds, ch04_flows.py, ch05_gans.py, ch14_policy.py, and ch15_memorization.py about half a minute each, ch02_autoregressive.py, ch09_flow.py, ch11_latent.py, and ch12_masked.py about a minute, ch06_score.py and ch08_sde.py about two minutes, ch03_vae.py about three, ch07_diffusion.py about four, and ch10_guidance.py about six. Training results are single runs; they illustrate the claims in the notes and are not benchmarks.

1. Foundations of generative modeling {#1-foundations-of-generative-modeling}

Script: ch01_foundations.py.

Local figure Data and construction

gen-manifold.png The digits at their original intensities, with the module’s split (NumPy seed 0). 297 samples from a Gaussian with the mean and sample covariance of the 1,500 training digits, drawn by NumPy’s multivariate_normal from the same generator. Left: the first eight held-out digits (top) and the first eight Gaussian samples (bottom), clipped to 0–16. Right: histograms, in 40 bins from 0 to 60, of the Euclidean distance from each held-out digit and each Gaussian sample to the nearest of the 1,500 training digits. The script also prints the two medians and the share of Gaussian samples closer than the 95th percentile of the held-out distances.

gen-divergences.png Left: the target \(p=0.6\,\mathcal N(-3,1)+0.4\,\mathcal N(3,1)\) on a grid of 24,001 points from −12 to 12, on which KL divergences are Riemann sums. The single Gaussian that minimizes the forward divergence \(\mathrm{KL}(p\,\Vert\,q)\) or the reverse divergence \(\mathrm{KL}(q\,\Vert\,p)\) is found by SciPy’s Nelder–Mead over the mean and the log standard deviation (tolerances \(10^{-6}\) and \(10^{-9}\), at most 4,000 iterations), starting from mean 0 for the forward fit and from means −1 and 1 for the reverse fit, all with standard deviation 1; the legend gives each fit’s mean and standard deviation. Right: for 121 shifts \(\theta\) from 0 to 3 between \(\mathcal N(0,0.05^2)\) and \(\mathcal N(\theta,0.05^2)\), the KL divergence \(\theta^2/(2\sigma^2)\), the Wasserstein-1 distance \(\theta\), and the MMD with a Gaussian kernel of width 1, all in closed form, and the Jensen–Shannon divergence in nats by a Riemann sum on 50,001 points from −1 to 4; the dotted line is \(\log2\), and the vertical axis is cut at 3.2. The script prints the four values at \(\theta=1\).

Local figure	Data and construction
gen-manifold.png	The digits at their original intensities, with the module’s split (NumPy seed 0). 297 samples from a Gaussian with the mean and sample covariance of the 1,500 training digits, drawn by NumPy’s `multivariate_normal` from the same generator. Left: the first eight held-out digits (top) and the first eight Gaussian samples (bottom), clipped to 0–16. Right: histograms, in 40 bins from 0 to 60, of the Euclidean distance from each held-out digit and each Gaussian sample to the nearest of the 1,500 training digits. The script also prints the two medians and the share of Gaussian samples closer than the 95th percentile of the held-out distances.
gen-divergences.png	Left: the target \(p=0.6\,\mathcal N(-3,1)+0.4\,\mathcal N(3,1)\) on a grid of 24,001 points from −12 to 12, on which KL divergences are Riemann sums. The single Gaussian that minimizes the forward divergence \(\mathrm{KL}(p\,\Vert\,q)\) or the reverse divergence \(\mathrm{KL}(q\,\Vert\,p)\) is found by SciPy’s Nelder–Mead over the mean and the log standard deviation (tolerances \(10^{-6}\) and \(10^{-9}\), at most 4,000 iterations), starting from mean 0 for the forward fit and from means −1 and 1 for the reverse fit, all with standard deviation 1; the legend gives each fit’s mean and standard deviation. Right: for 121 shifts \(\theta\) from 0 to 3 between \(\mathcal N(0,0.05^2)\) and \(\mathcal N(\theta,0.05^2)\), the KL divergence \(\theta^2/(2\sigma^2)\), the Wasserstein-1 distance \(\theta\), and the MMD with a Gaussian kernel of width 1, all in closed form, and the Jensen–Shannon divergence in nats by a Riemann sum on 50,001 points from −1 to 4; the dotted line is \(\log2\), and the vertical axis is cut at 3.2. The script prints the four values at \(\theta=1\).

2. Autoregressive models {#2-autoregressive-models}

Script: ch02_autoregressive.py.

Local figure	Data and construction
gen-ar-made.png	The binarized digits with the module’s split (NumPy seed 0). MADE with 64 inputs, two hidden layers of 256 ReLU units, and 64 output logits, its weights masked by degrees: hidden degrees are drawn uniformly from 1 to 63 by the same NumPy generator; a first-layer unit sees the inputs whose position in the ordering is at most its degree, a second-layer unit sees the first-layer units whose degree is at most its own, and output \(d\) sees the second-layer units whose degree is below the position of \(d\). Two models, one in raster order and one in a random order drawn from the NumPy generator, each trained for 200 epochs of shuffled minibatches of 100 with Adam (learning rate \(10^{-3}\), weight decay \(10^{-4}\)) on the binary cross-entropy (torch seed 0). Left: the first 12 held-out digits and 12 of 24 samples from the random-order model, drawn one pixel at a time in its order. Right: bits per pixel on the training (dashed) and held-out (solid) digits after each epoch, and the held-out bits per pixel of independent pixels with the training frequencies clipped to \([0.001,0.999]\) (dotted). The script also prints the final training and held-out values and the best held-out value of each model.
gen-ar-receptive.png	No data are used. Left: 17 positions in five rows, the input and four causal convolutions of kernel size 2 with dilations 1, 2, 4, and 8, in which each unit depends on the unit below it and on the one \(d\) positions earlier; the dependencies of the last output are traced back in blue and reach 16 inputs. Right: the row of the pixel being predicted (red) and the five rows above it, five columns on either side. The visible set starts from the \(3\times3\) mask-A neighborhood (the three pixels above and the one to the left), grows through four \(3\times3\) mask-B layers, which also include the center, and is restricted to earlier pixels in raster order: 50 visible earlier pixels are blue, the 10 earlier pixels of the window that stay invisible are light orange, and later pixels are white.

gen-ar-made.png

The binarized digits with the module’s split (NumPy seed 0). MADE with 64 inputs, two hidden layers of 256 ReLU units, and 64 output logits, its weights masked by degrees: hidden degrees are drawn uniformly from 1 to 63 by the same NumPy generator; a first-layer unit sees the inputs whose position in the ordering is at most its degree, a second-layer unit sees the first-layer units whose degree is at most its own, and output \(d\) sees the second-layer units whose degree is below the position of \(d\). Two models, one in raster order and one in a random order drawn from the NumPy generator, each trained for 200 epochs of shuffled minibatches of 100 with Adam (learning rate \(10^{-3}\), weight decay \(10^{-4}\)) on the binary cross-entropy (torch seed 0). Left: the first 12 held-out digits and 12 of 24 samples from the random-order model, drawn one pixel at a time in its order. Right: bits per pixel on the training (dashed) and held-out (solid) digits after each epoch, and the held-out bits per pixel of independent pixels with the training frequencies clipped to \([0.001,0.999]\) (dotted). The script also prints the final training and held-out values and the best held-out value of each model.

gen-ar-receptive.png

No data are used. Left: 17 positions in five rows, the input and four causal convolutions of kernel size 2 with dilations 1, 2, 4, and 8, in which each unit depends on the unit below it and on the one \(d\) positions earlier; the dependencies of the last output are traced back in blue and reach 16 inputs. Right: the row of the pixel being predicted (red) and the five rows above it, five columns on either side. The visible set starts from the \(3\times3\) mask-A neighborhood (the three pixels above and the one to the left), grows through four \(3\times3\) mask-B layers, which also include the center, and is restricted to earlier pixels in raster order: 50 visible earlier pixels are blue, the 10 earlier pixels of the window that stay invisible are light orange, and later pixels are white.

3. Variational autoencoders {#3-variational-autoencoders}

Script: ch03_vae.py. Both figures use VAEs of the binarized digits with the module’s split (NumPy seed 0): a diagonal Gaussian encoder with two hidden layers of 256 ReLU units, a decoder of the same shape with Bernoulli logits, and a standard normal prior. Each model is trained for 300 epochs of shuffled minibatches of 100 with Adam at learning rate \(10^{-3}\) on distortion plus \(\beta\) times rate, with one reparameterized sample per digit; the models are trained in sequence after one torch seed 0. Rate is the closed-form KL divergence from the prior and distortion the binary cross-entropy of one sample, both reported in bits per image.

Local figure	Data and construction
gen-vae-latent.png	Two latent dimensions and \(\beta=1\). Left: the encoder means of the 297 held-out digits, colored by label with the tab10 colormap, with each class’s numeral at its median. Right: the decoder’s pixel probabilities for latent codes on a \(12\times12\) grid of 12 values from −2.5 to 2.5 in each coordinate, dark for probability 1. The script prints the held-out distortion and rate.
gen-vae-rate.png	Eight latent dimensions and \(\beta=0.25\), 0.5, 1, 2, 4, and 8. Left: distortion against rate on the 1,500 training digits (solid) and the 297 held-out digits (dashed), each estimated with one sample per digit; the gray dashed line of slope −1 marks the lowest held-out sum of rate and distortion. Right: the first six held-out digits and the decoder’s probabilities at their encoder means for \(\beta=0.25\), 1, 4, and 8. The script prints the four values and the held-out negative ELBO for each \(\beta\).

gen-vae-latent.png

Two latent dimensions and \(\beta=1\). Left: the encoder means of the 297 held-out digits, colored by label with the tab10 colormap, with each class’s numeral at its median. Right: the decoder’s pixel probabilities for latent codes on a \(12\times12\) grid of 12 values from −2.5 to 2.5 in each coordinate, dark for probability 1. The script prints the held-out distortion and rate.

gen-vae-rate.png

Eight latent dimensions and \(\beta=0.25\), 0.5, 1, 2, 4, and 8. Left: distortion against rate on the 1,500 training digits (solid) and the 297 held-out digits (dashed), each estimated with one sample per digit; the gray dashed line of slope −1 marks the lowest held-out sum of rate and distortion. Right: the first six held-out digits and the decoder’s probabilities at their encoder means for \(\beta=0.25\), 1, 4, and 8. The script prints the four values and the held-out negative ELBO for each \(\beta\).

4. Normalizing flows {#4-normalizing-flows}

Script: ch04_flows.py.

Local figure Data and construction

gen-flow-moons.png 3,000 points from scikit-learn’s make_moons (noise 0.06, random_state=0), standardized per coordinate; the first 2,000 for training and the last 1,000 held out. Eight affine coupling layers that alternate the coordinate they keep; each maps the other coordinate \(b\) to \(b\,e^{s}+t\), with \(s\) and \(t\) computed from the kept coordinate by an MLP with two hidden layers of 64 ReLU units and \(s\) passed through tanh; the base is a standard bivariate normal (torch seed 0). Adam with learning rate \(2\times10^{-3}\) on the negative log-likelihood, 3,000 steps of 256 points drawn with replacement. Panels: all 3,000 points; the learned density on a \(300\times300\) grid over \([-2.6,2.6]^2\); 2,000 samples pushed from Gaussian noise (generator seed 1); and the images of 11 vertical (blue) and 11 horizontal (orange) latent lines at −2.5, −2, …, 2.5, each of 200 points from −2.5 to 2.5. The script prints the held-out log-likelihood and the shares of samples and of held-out points farther than 0.25 from every training point.

Local figure	Data and construction
gen-flow-moons.png	3,000 points from scikit-learn’s `make_moons` (noise 0.06, `random_state=0`), standardized per coordinate; the first 2,000 for training and the last 1,000 held out. Eight affine coupling layers that alternate the coordinate they keep; each maps the other coordinate \(b\) to \(b\,e^{s}+t\), with \(s\) and \(t\) computed from the kept coordinate by an MLP with two hidden layers of 64 ReLU units and \(s\) passed through tanh; the base is a standard bivariate normal (torch seed 0). Adam with learning rate \(2\times10^{-3}\) on the negative log-likelihood, 3,000 steps of 256 points drawn with replacement. Panels: all 3,000 points; the learned density on a \(300\times300\) grid over \([-2.6,2.6]^2\); 2,000 samples pushed from Gaussian noise (generator seed 1); and the images of 11 vertical (blue) and 11 horizontal (orange) latent lines at −2.5, −2, …, 2.5, each of 200 points from −2.5 to 2.5. The script prints the held-out log-likelihood and the shares of samples and of held-out points farther than 0.25 from every training point.

5. Generative adversarial networks {#5-generative-adversarial-networks}

Script: ch05_gans.py.

Local figure	Data and construction
gen-gan-dirac.png	No data are used. The Dirac GAN: data at 0, a generator that puts all its mass at \(\theta\), and a discriminator with logit \(\psi x\). From \((\theta,\psi)=(1,1)\) with step size \(h=0.1\), 2,000 steps of \(\theta\leftarrow\theta+h\,\psi\,\sigma(\psi\theta)\) and \(\psi\leftarrow\psi-h\,\bigl(\theta\,\sigma(\psi\theta)+\gamma\psi\bigr)\), applied simultaneously with \(\gamma=0\) (red), alternately with the discriminator first and \(\gamma=0\) (blue), and simultaneously with the R1 penalty \(\gamma=0.3\) (green). Left: the first 800 steps in the \((\theta,\psi)\) plane, with the start and the equilibrium (star). Right: the distance from \((0,0)\) at every step on a logarithmic axis from \(10^{-6}\) to 10. The script prints the distances after 2,000 steps.
gen-gan-ring.png	Eight Gaussians of standard deviation 0.05 centered on a circle of radius 2, chosen with equal probability and sampled afresh at every step (torch seed 0). Generator and discriminator are MLPs with two hidden layers of 128 ReLU units, from two-dimensional standard normal noise to a point and from a point to a logit. Adam with learning rate \(10^{-3}\) and betas \((0.5,0.999)\) for both; each of 4,000 steps takes 256 real and 256 generated points, updates the discriminator on the binary cross-entropy, and then updates the generator on the non-saturating loss with the same generated points. At steps 250, 1,000, and 4,000: 1,500 of 10,000 generator samples (dark), the modes (orange circles), and the discriminator’s probability of real on a \(200\times200\) grid over \([-3,3]^2\) (RdBu colormap from 0 to 1 at 55% opacity).

gen-gan-dirac.png

No data are used. The Dirac GAN: data at 0, a generator that puts all its mass at \(\theta\), and a discriminator with logit \(\psi x\). From \((\theta,\psi)=(1,1)\) with step size \(h=0.1\), 2,000 steps of \(\theta\leftarrow\theta+h\,\psi\,\sigma(\psi\theta)\) and \(\psi\leftarrow\psi-h\,\bigl(\theta\,\sigma(\psi\theta)+\gamma\psi\bigr)\), applied simultaneously with \(\gamma=0\) (red), alternately with the discriminator first and \(\gamma=0\) (blue), and simultaneously with the R1 penalty \(\gamma=0.3\) (green). Left: the first 800 steps in the \((\theta,\psi)\) plane, with the start and the equilibrium (star). Right: the distance from \((0,0)\) at every step on a logarithmic axis from \(10^{-6}\) to 10. The script prints the distances after 2,000 steps.

gen-gan-ring.png

Eight Gaussians of standard deviation 0.05 centered on a circle of radius 2, chosen with equal probability and sampled afresh at every step (torch seed 0). Generator and discriminator are MLPs with two hidden layers of 128 ReLU units, from two-dimensional standard normal noise to a point and from a point to a logit. Adam with learning rate \(10^{-3}\) and betas \((0.5,0.999)\) for both; each of 4,000 steps takes 256 real and 256 generated points, updates the discriminator on the binary cross-entropy, and then updates the generator on the non-saturating loss with the same generated points. At steps 250, 1,000, and 4,000: 1,500 of 10,000 generator samples (dark), the modes (orange circles), and the discriminator’s probability of real on a \(200\times200\) grid over \([-3,3]^2\) (RdBu colormap from 0 to 1 at 55% opacity).

6. Energy-based models and score matching {#6-energy-based-models-and-score-matching}

Script: ch06_score.py.

Local figure	Data and construction
gen-score-annealing.png	Eight Gaussians of standard deviation 0.1 on a circle of radius 2, mode \(k=1,\dots,8\) at angle \(2\pi(k-1)/8\) with weight \(k/36\) (torch seed 0). Twelve noise levels geometrically spaced from 3 to 0.05. The network maps \((x,\log\sigma)\) through three hidden layers of 256 SiLU units to two outputs, and the score is its output divided by \(\sigma\). Denoising score matching with weight \(\sigma^2\) (the output regressed on minus the added noise), with a level drawn uniformly for each example, Adam with learning rate \(10^{-3}\), and 8,000 steps of 512 fresh samples. First two panels: the learned score, normalized to unit length, on a \(17\times17\) grid over \([-3,3]^2\) at \(\sigma=0.05\) and at the fourth level, \(\sigma=0.98\), with orange disks of area proportional to the weights. Last two panels: Langevin dynamics \(x\leftarrow x+\tfrac{\epsilon}{2}\,s(x,\sigma)+\sqrt{\epsilon}\,z\) with \(\epsilon=0.1\,\sigma^2\) from the same 4,000 points uniform on \([-3,3]^2\) (generator seed 1), for 1,200 steps at \(\sigma=0.05\) alone and for 100 steps at each of the 12 levels in decreasing order. The script prints, for both samplers, the share of samples within 0.3 of a mode, the shares of the modes among them, and their total variation from the weights.

gen-score-annealing.png

Eight Gaussians of standard deviation 0.1 on a circle of radius 2, mode \(k=1,\dots,8\) at angle \(2\pi(k-1)/8\) with weight \(k/36\) (torch seed 0). Twelve noise levels geometrically spaced from 3 to 0.05. The network maps \((x,\log\sigma)\) through three hidden layers of 256 SiLU units to two outputs, and the score is its output divided by \(\sigma\). Denoising score matching with weight \(\sigma^2\) (the output regressed on minus the added noise), with a level drawn uniformly for each example, Adam with learning rate \(10^{-3}\), and 8,000 steps of 512 fresh samples. First two panels: the learned score, normalized to unit length, on a \(17\times17\) grid over \([-3,3]^2\) at \(\sigma=0.05\) and at the fourth level, \(\sigma=0.98\), with orange disks of area proportional to the weights. Last two panels: Langevin dynamics \(x\leftarrow x+\tfrac{\epsilon}{2}\,s(x,\sigma)+\sqrt{\epsilon}\,z\) with \(\epsilon=0.1\,\sigma^2\) from the same 4,000 points uniform on \([-3,3]^2\) (generator seed 1), for 1,200 steps at \(\sigma=0.05\) alone and for 100 steps at each of the 12 levels in decreasing order. The script prints, for both samplers, the share of samples within 0.3 of a mode, the shares of the modes among them, and their total variation from the weights.

7. Denoising diffusion models {#7-denoising-diffusion-models}

Script: ch07_diffusion.py. Both figures use \(T=1000\) and the digits scaled to \([-1,1]\) with the module’s split (NumPy seed 0). The linear schedule has \(\beta_t\) evenly spaced from \(10^{-4}\) to 0.02; the cosine schedule has \(\beta_t=1-f(t)/f(t-1)\), clipped to \([0,0.999]\), with \(f(t)=\cos^2\bigl(\tfrac{t/T+s}{1+s}\cdot\tfrac{\pi}{2}\bigr)\) and \(s=0.008\).

Local figure Data and construction

gen-diffusion-forward.png Left: the first held-out 3, noised as \(\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon\) with one draw of \(\epsilon\) (NumPy seed 3) at \(t=0\), 100, 200, 300, 500, 700, and 1000 under the linear schedule (top) and the cosine schedule (bottom), on a gray scale from −2 to 2. Right: \(\log_{10}\bigl(\bar\alpha_t/(1-\bar\alpha_t)\bigr)\) for \(t=1,\dots,1000\) under both schedules, with a dotted line at 0 and the axis limited to −9.5 to 5.

gen-diffusion-digits.png A DDPM with the linear schedule on the 1,500 training digits (torch seed 0). The noise predictor is an MLP from the 64 pixels and a 32-dimensional sinusoidal embedding of the step (sines and cosines of \(t\,e^{-k\ln1000/16}\), \(k=0,\dots,15\)) through three hidden layers of 512 SiLU units to 64 outputs. Adam with learning rate \(10^{-3}\) on \(L_{\text{simple}}\), 15,000 steps of 256 digits drawn with replacement, each at a step drawn uniformly, and an exponential moving average of the weights with decay 0.999, which is used for sampling. 1,000 samples by ancestral sampling with variance \(\beta_t\) from Gaussian noise (generator seed 1), clipped to \([-1,1]\). Left: the first 36 samples. Right: for the first two samples, the state \(x_t\) (gray scale from −2.5 to 2.5) and the prediction \(\hat x_0=(x_t-\sqrt{1-\bar\alpha_t}\,\hat\epsilon)/\sqrt{\bar\alpha_t}\), clipped to \([-1,1]\), at steps 1000, 800, 600, 400, 250, 100, and 1. The script also fits scikit-learn’s logistic regression (C=0.1, at most 5,000 iterations) to the training digits and prints its held-out accuracy, its mean top-class probability on the samples and on the held-out digits, the number of samples predicted in each class, and the distances from samples and held-out digits to their nearest training image.

Local figure	Data and construction
gen-diffusion-forward.png	Left: the first held-out 3, noised as \(\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon\) with one draw of \(\epsilon\) (NumPy seed 3) at \(t=0\), 100, 200, 300, 500, 700, and 1000 under the linear schedule (top) and the cosine schedule (bottom), on a gray scale from −2 to 2. Right: \(\log_{10}\bigl(\bar\alpha_t/(1-\bar\alpha_t)\bigr)\) for \(t=1,\dots,1000\) under both schedules, with a dotted line at 0 and the axis limited to −9.5 to 5.
gen-diffusion-digits.png	A DDPM with the linear schedule on the 1,500 training digits (torch seed 0). The noise predictor is an MLP from the 64 pixels and a 32-dimensional sinusoidal embedding of the step (sines and cosines of \(t\,e^{-k\ln1000/16}\), \(k=0,\dots,15\)) through three hidden layers of 512 SiLU units to 64 outputs. Adam with learning rate \(10^{-3}\) on \(L_{\text{simple}}\), 15,000 steps of 256 digits drawn with replacement, each at a step drawn uniformly, and an exponential moving average of the weights with decay 0.999, which is used for sampling. 1,000 samples by ancestral sampling with variance \(\beta_t\) from Gaussian noise (generator seed 1), clipped to \([-1,1]\). Left: the first 36 samples. Right: for the first two samples, the state \(x_t\) (gray scale from −2.5 to 2.5) and the prediction \(\hat x_0=(x_t-\sqrt{1-\bar\alpha_t}\,\hat\epsilon)/\sqrt{\bar\alpha_t}\), clipped to \([-1,1]\), at steps 1000, 800, 600, 400, 250, 100, and 1. The script also fits scikit-learn’s logistic regression (`C=0.1`, at most 5,000 iterations) to the training digits and prints its held-out accuracy, its mean top-class probability on the samples and on the held-out digits, the number of samples predicted in each class, and the distances from samples and held-out digits to their nearest training image.

8. Diffusion SDEs and fast sampling {#8-diffusion-sdes-and-fast-sampling}

Script: ch08_sde.py.

Local figure	Data and construction
gen-sde-paths.png	The mixture \(0.8\,\mathcal N(-4,0.5^2)+0.2\,\mathcal N(4,0.5^2)\) under the VP SDE with \(\beta(t)=0.1+19.9\,t\) and \(\bar\alpha(t)=e^{-(0.1t+9.95t^2)}\), with the exact score of the noised mixture. Time runs from 1 to 0.001 in 1,000 equal steps. Left: 40 reverse-time SDE paths by Euler–Maruyama from standard normal starting points (NumPy seed 2, which also gives the increments). Right: 40 probability-flow ODE paths by Heun’s method from the quantiles \((k+0.5)/40\), \(k=0,\dots,39\), of the standard normal. Paths ending below 0 are dark and the others orange; the background is the square root of the density of \(x_t\) on a \(400\times400\) grid with \(x\) from −6.5 to 6.5, in the Blues colormap. The script prints the share of paths of each kind that end in the left mode.
gen-sde-distillation.png	The eight Gaussians of chapter 6 (circle of radius 2, weights 1/36 to 8/36, standard deviation 0.1), sampled afresh for every batch (torch seed 0). Denoiser \(D(x,\sigma)=c_{\text{skip}}\,x+c_{\text{out}}\,F\bigl(c_{\text{in}}\,x,\tfrac14\ln\sigma\bigr)\) with the EDM coefficients for \(\sigma_{\text{data}}=1.4\) and \(F\) an MLP with three hidden layers of 256 SiLU units, trained by regressing \(F\) on \((\text{target}-c_{\text{skip}}\,z)/c_{\text{out}}\) for the noisy point \(z\), with Adam at learning rate \(10^{-3}\), batches of 512, and \(\sigma\) drawn uniformly from the nonzero levels of the current schedule. Teacher schedule: 64 levels \(\bigl(\sigma_{\max}^{1/\rho}+\tfrac{i}{63}(\sigma_{\min}^{1/\rho}-\sigma_{\max}^{1/\rho})\bigr)^{\rho}\), \(i=0,\dots,63\), with \(\sigma_{\max}=80\), \(\sigma_{\min}=0.002\), and \(\rho=7\), followed by 0. The base model is trained for 4,000 steps with the clean point as target. Each of six distillation rounds copies the current model and trains it for 1,500 steps on every second level of its schedule, with the target that makes one DDIM step between consecutive new levels land where two DDIM steps of the previous model, through the skipped level, land; the rounds halve the steps from 64 to 1. Sampling is deterministic DDIM, \(x\leftarrow x+(\sigma'-\sigma)\,(x-D(x,\sigma))/\sigma\), from the same 4,000 starting points \(80\,z\) (generator seed 1) on the teacher schedule thinned to \(n\) steps. Top: the base model with 64, 8, 2, and 1 steps. Bottom: 4,000 data points and the students with 8, 2, and 1 steps. Titles give the share of samples within 0.3 of a mode, and orange circles are sized by weight. The script prints these shares for 64 down to 1 steps and the total variation between the mode shares and the weights.

gen-sde-paths.png

The mixture \(0.8\,\mathcal N(-4,0.5^2)+0.2\,\mathcal N(4,0.5^2)\) under the VP SDE with \(\beta(t)=0.1+19.9\,t\) and \(\bar\alpha(t)=e^{-(0.1t+9.95t^2)}\), with the exact score of the noised mixture. Time runs from 1 to 0.001 in 1,000 equal steps. Left: 40 reverse-time SDE paths by Euler–Maruyama from standard normal starting points (NumPy seed 2, which also gives the increments). Right: 40 probability-flow ODE paths by Heun’s method from the quantiles \((k+0.5)/40\), \(k=0,\dots,39\), of the standard normal. Paths ending below 0 are dark and the others orange; the background is the square root of the density of \(x_t\) on a \(400\times400\) grid with \(x\) from −6.5 to 6.5, in the Blues colormap. The script prints the share of paths of each kind that end in the left mode.

gen-sde-distillation.png

The eight Gaussians of chapter 6 (circle of radius 2, weights 1/36 to 8/36, standard deviation 0.1), sampled afresh for every batch (torch seed 0). Denoiser \(D(x,\sigma)=c_{\text{skip}}\,x+c_{\text{out}}\,F\bigl(c_{\text{in}}\,x,\tfrac14\ln\sigma\bigr)\) with the EDM coefficients for \(\sigma_{\text{data}}=1.4\) and \(F\) an MLP with three hidden layers of 256 SiLU units, trained by regressing \(F\) on \((\text{target}-c_{\text{skip}}\,z)/c_{\text{out}}\) for the noisy point \(z\), with Adam at learning rate \(10^{-3}\), batches of 512, and \(\sigma\) drawn uniformly from the nonzero levels of the current schedule. Teacher schedule: 64 levels \(\bigl(\sigma_{\max}^{1/\rho}+\tfrac{i}{63}(\sigma_{\min}^{1/\rho}-\sigma_{\max}^{1/\rho})\bigr)^{\rho}\), \(i=0,\dots,63\), with \(\sigma_{\max}=80\), \(\sigma_{\min}=0.002\), and \(\rho=7\), followed by 0. The base model is trained for 4,000 steps with the clean point as target. Each of six distillation rounds copies the current model and trains it for 1,500 steps on every second level of its schedule, with the target that makes one DDIM step between consecutive new levels land where two DDIM steps of the previous model, through the skipped level, land; the rounds halve the steps from 64 to 1. Sampling is deterministic DDIM, \(x\leftarrow x+(\sigma'-\sigma)\,(x-D(x,\sigma))/\sigma\), from the same 4,000 starting points \(80\,z\) (generator seed 1) on the teacher schedule thinned to \(n\) steps. Top: the base model with 64, 8, 2, and 1 steps. Bottom: 4,000 data points and the students with 8, 2, and 1 steps. Titles give the share of samples within 0.3 of a mode, and orange circles are sized by weight. The script prints these shares for 64 down to 1 steps and the total variation between the mode shares and the weights.

9. Flow matching {#9-flow-matching}

Script: ch09_flow.py.

Local figure Data and construction

gen-flow-matching.png The eight Gaussians of chapter 6 (circle of radius 2, weights 1/36 to 8/36, standard deviation 0.1), sampled afresh for every batch. Each velocity field is an MLP from \((x,t)\) through three hidden layers of 128 SiLU units to two outputs, trained with Adam at learning rate \(10^{-3}\) for 3,000 steps of 256 pairs on straight paths \(x_t=(1-t)\,x_0+t\,x_1\), with \(t\) uniform on \([0,1]\) and target \(x_1-x_0\). Couplings: independent pairs of standard normal noise and data (torch seed 0); minibatch optimal transport, pairing each batch by SciPy’s linear_sum_assignment on squared distances (torch seed 0); and reflow, a network initialized with the first flow’s weights and trained (torch seed 0) on pairs drawn from 50,000 noise points and their images under the first flow with 100 Euler steps. Top: straight lines between 70 noise points (generator seed 5) and 70 data points, then the paths of the flows with independent coupling, after reflow, and with minibatch optimal transport from the same 70 noise points with 100 Euler steps. Bottom: 4,000 data points, and the samples of each flow after one Euler step from 4,000 starting points (generator seed 1), with the share within 0.3 of a mode in the titles. The script also prints the shares with 2 and 100 steps and the median distance between the endpoints of 1 and 100 steps.

Local figure	Data and construction
gen-flow-matching.png	The eight Gaussians of chapter 6 (circle of radius 2, weights 1/36 to 8/36, standard deviation 0.1), sampled afresh for every batch. Each velocity field is an MLP from \((x,t)\) through three hidden layers of 128 SiLU units to two outputs, trained with Adam at learning rate \(10^{-3}\) for 3,000 steps of 256 pairs on straight paths \(x_t=(1-t)\,x_0+t\,x_1\), with \(t\) uniform on \([0,1]\) and target \(x_1-x_0\). Couplings: independent pairs of standard normal noise and data (torch seed 0); minibatch optimal transport, pairing each batch by SciPy’s `linear_sum_assignment` on squared distances (torch seed 0); and reflow, a network initialized with the first flow’s weights and trained (torch seed 0) on pairs drawn from 50,000 noise points and their images under the first flow with 100 Euler steps. Top: straight lines between 70 noise points (generator seed 5) and 70 data points, then the paths of the flows with independent coupling, after reflow, and with minibatch optimal transport from the same 70 noise points with 100 Euler steps. Bottom: 4,000 data points, and the samples of each flow after one Euler step from 4,000 starting points (generator seed 1), with the share within 0.3 of a mode in the titles. The script also prints the shares with 2 and 100 steps and the median distance between the endpoints of 1 and 100 steps.

10. Guidance and conditional generation {#10-guidance-and-conditional-generation}

Script: ch10_guidance.py. Both figures use one class-conditional DDPM of the digits scaled to \([-1,1]\) with the module’s split (NumPy seed 0), with the linear schedule of chapter 7 (\(T=1000\), \(\beta_t\) from \(10^{-4}\) to 0.02). The noise predictor maps the 64 pixels, the sinusoidal step embedding of chapter 7, and a learned 32-dimensional embedding of the label, with an eleventh entry for no label, through three hidden layers of 512 SiLU units to 64 outputs; during training the label is replaced by the null entry with probability 0.1. Adam with learning rate \(10^{-3}\), 15,000 steps of 256 digits drawn with replacement, and an exponential moving average of the weights with decay 0.999, which is used for sampling (torch seed 0). The classifier is scikit-learn’s logistic regression (C=0.1, at most 5,000 iterations) fitted to the training digits.

Local figure	Data and construction
gen-guidance-digits.png	50 samples of each class with classifier-free guidance, \(\hat\epsilon=(1+w)\,\hat\epsilon(x_t,c)-w\,\hat\epsilon(x_t,\varnothing)\), at \(w=0\), 1, and 3, by ancestral sampling from the same noise (generator seed 1 for each \(w\)), clipped to \([-1,1]\). Each panel shows the first six samples of each class, one class per row. The script prints, for each \(w\) and for the held-out digits, the classifier’s agreement with the label, its mean probability of the label, the share of pixels that differ between two samples of the same class binarized at 0 (averaged over all pairs within a class, a sample with itself included, and over classes), and the share of pixels beyond \(\pm0.9\).
gen-guidance-inpainting.png	The upper half (rows 1–4, 32 pixels) of each of the first 200 held-out digits is observed, and three completions of each are sampled with the unconditional (null-label) predictions by each method (generator seed 2 for each method). Replacement: after each ancestral step, the observed pixels are replaced by the observation \(y\) noised to the new step’s level, \(\sqrt{\bar\alpha_{t-1}}\,y+\sqrt{1-\bar\alpha_{t-1}}\,z\), and by the observation itself after the last step. DPS: each ancestral step is followed by a move of \(-\zeta\,\nabla_{x_t}r^2/r\), where \(r\) is the norm of the difference between \(y\) and \(\hat x_0\) on the observed pixels and \(\zeta=1\). Rows: the first six held-out digits; columns: the original, the observed half with the missing half in light blue, three replacement completions, and three DPS completions, with an orange line at the boundary. The script prints, for each method, the classifier’s agreement with the true label, the squared error on the missing and on the observed half, and the standard deviation across the three completions.

gen-guidance-digits.png

50 samples of each class with classifier-free guidance, \(\hat\epsilon=(1+w)\,\hat\epsilon(x_t,c)-w\,\hat\epsilon(x_t,\varnothing)\), at \(w=0\), 1, and 3, by ancestral sampling from the same noise (generator seed 1 for each \(w\)), clipped to \([-1,1]\). Each panel shows the first six samples of each class, one class per row. The script prints, for each \(w\) and for the held-out digits, the classifier’s agreement with the label, its mean probability of the label, the share of pixels that differ between two samples of the same class binarized at 0 (averaged over all pairs within a class, a sample with itself included, and over classes), and the share of pixels beyond \(\pm0.9\).

gen-guidance-inpainting.png

The upper half (rows 1–4, 32 pixels) of each of the first 200 held-out digits is observed, and three completions of each are sampled with the unconditional (null-label) predictions by each method (generator seed 2 for each method). Replacement: after each ancestral step, the observed pixels are replaced by the observation \(y\) noised to the new step’s level, \(\sqrt{\bar\alpha_{t-1}}\,y+\sqrt{1-\bar\alpha_{t-1}}\,z\), and by the observation itself after the last step. DPS: each ancestral step is followed by a move of \(-\zeta\,\nabla_{x_t}r^2/r\), where \(r\) is the norm of the difference between \(y\) and \(\hat x_0\) on the observed pixels and \(\zeta=1\). Rows: the first six held-out digits; columns: the original, the observed half with the missing half in light blue, three replacement completions, and three DPS completions, with an orange line at the boundary. The script prints, for each method, the classifier’s agreement with the true label, the squared error on the missing and on the observed half, and the standard deviation across the three completions.

11. Latent diffusion and large-scale generation {#11-latent-diffusion-and-large-scale-generation}

Script: ch11_latent.py.

Local figure Data and construction

gen-latent-diffusion.png The digits scaled to \([-1,1]\) with the module’s split (NumPy seed 0; torch seed 0). VAE: an encoder with two hidden layers of 256 SiLU units giving the mean and log variance of 8 latent numbers, and a decoder with two hidden layers of 256 SiLU units and a tanh output; the loss is the squared error summed over pixels plus 0.01 times the KL divergence from the standard normal, minimized by Adam with learning rate \(10^{-3}\) for 3,000 steps of 256 digits drawn with replacement. The encoder means of the training digits, divided by their overall standard deviation, are the data of a DDPM with the linear schedule of chapter 7, whose noise predictor maps the latent and the sinusoidal step embedding of chapter 7 through three hidden layers of 256 SiLU units to 8 outputs, trained with Adam at learning rate \(10^{-3}\) for 6,000 steps of 256 latents without weight averaging. Left: the first 18 held-out digits, each above its decoding from the encoder mean. Middle: 36 ancestral samples (generator seed 1), rescaled and decoded. Right: scikit-learn’s logistic regression (C=0.1, at most 5,000 iterations), fitted to the training digits, applied to the 297 held-out digits upsampled by pixel repetition to \(8k\times8k\) for \(k=1\), 2, 4, and 8, noised as \(\sqrt{\bar\alpha}\,x+\sqrt{1-\bar\alpha}\,\epsilon\) at 19 per-pixel \(\log_{10}\) signal-to-noise ratios \(\log_{10}\bigl(\bar\alpha/(1-\bar\alpha)\bigr)\) from 1 down to −3.5, average-pooled back to \(8\times8\), and divided by \(\sqrt{\bar\alpha}\); each point averages 10 noise draws (NumPy seed 0), and the dotted line is chance, 0.1.

Local figure	Data and construction
gen-latent-diffusion.png	The digits scaled to \([-1,1]\) with the module’s split (NumPy seed 0; torch seed 0). VAE: an encoder with two hidden layers of 256 SiLU units giving the mean and log variance of 8 latent numbers, and a decoder with two hidden layers of 256 SiLU units and a tanh output; the loss is the squared error summed over pixels plus 0.01 times the KL divergence from the standard normal, minimized by Adam with learning rate \(10^{-3}\) for 3,000 steps of 256 digits drawn with replacement. The encoder means of the training digits, divided by their overall standard deviation, are the data of a DDPM with the linear schedule of chapter 7, whose noise predictor maps the latent and the sinusoidal step embedding of chapter 7 through three hidden layers of 256 SiLU units to 8 outputs, trained with Adam at learning rate \(10^{-3}\) for 6,000 steps of 256 latents without weight averaging. Left: the first 18 held-out digits, each above its decoding from the encoder mean. Middle: 36 ancestral samples (generator seed 1), rescaled and decoded. Right: scikit-learn’s logistic regression (`C=0.1`, at most 5,000 iterations), fitted to the training digits, applied to the 297 held-out digits upsampled by pixel repetition to \(8k\times8k\) for \(k=1\), 2, 4, and 8, noised as \(\sqrt{\bar\alpha}\,x+\sqrt{1-\bar\alpha}\,\epsilon\) at 19 per-pixel \(\log_{10}\) signal-to-noise ratios \(\log_{10}\bigl(\bar\alpha/(1-\bar\alpha)\bigr)\) from 1 down to −3.5, average-pooled back to \(8\times8\), and divided by \(\sqrt{\bar\alpha}\); each point averages 10 noise draws (NumPy seed 0), and the dotted line is chance, 0.1.

12. Discrete tokens and multimodal generation {#12-discrete-tokens-and-multimodal-generation}

Script: ch12_masked.py.

Local figure	Data and construction
gen-masked-diffusion.png	The 1,500 binarized training digits of the module’s split (NumPy seed 0; torch seed 0). The network maps the digit with its masked pixels set to 0, concatenated with the 64-pixel mask, through two hidden layers of 512 SiLU units to 64 logits of each pixel being on. Training: a time \(t\) uniform on \([0,1]\) (at least 0.001) per digit, each pixel masked independently with probability \(t\), and the binary cross-entropy summed over the masked pixels and divided by \(t\); Adam with learning rate \(10^{-3}\), 4,000 steps of 256 digits drawn with replacement. Sampling starts fully masked, and each of \(n\) steps, from \(t=s/n\) to \(t'=(s-1)/n\), draws a value for every pixel from the model’s probabilities and keeps it for the pixels it reveals: the random sampler reveals each masked pixel with probability \((t-t')/t\), and the confident sampler reveals the 8 masked pixels per step whose drawn values have the highest probability. Top: 36 samples each with 1, 4, and 64 random steps and 8 confident steps, drawn in turn from one generator (seed 1). Bottom: three samples with 64 random steps (generator seed 4) after 8, 20, 32, 48, and 64 steps, with masked pixels in light blue.

gen-masked-diffusion.png

The 1,500 binarized training digits of the module’s split (NumPy seed 0; torch seed 0). The network maps the digit with its masked pixels set to 0, concatenated with the 64-pixel mask, through two hidden layers of 512 SiLU units to 64 logits of each pixel being on. Training: a time \(t\) uniform on \([0,1]\) (at least 0.001) per digit, each pixel masked independently with probability \(t\), and the binary cross-entropy summed over the masked pixels and divided by \(t\); Adam with learning rate \(10^{-3}\), 4,000 steps of 256 digits drawn with replacement. Sampling starts fully masked, and each of \(n\) steps, from \(t=s/n\) to \(t'=(s-1)/n\), draws a value for every pixel from the model’s probabilities and keeps it for the pixels it reveals: the random sampler reveals each masked pixel with probability \((t-t')/t\), and the confident sampler reveals the 8 masked pixels per step whose drawn values have the highest probability. Top: 36 samples each with 1, 4, and 64 random steps and 8 confident steps, drawn in turn from one generator (seed 1). Bottom: three samples with 64 random steps (generator seed 4) after 8, 20, 32, 48, and 64 steps, with masked pixels in light blue.

13. Evaluating generative models {#13-evaluating-generative-models}

Script: ch13_eval.py.

Local figure Data and construction

gen-eval-precision-recall.png The eight Gaussians of chapter 6 (circle of radius 2, weights 1/36 to 8/36, standard deviation 0.1), with all draws from one NumPy generator (seed 0). The data are 2,000 points; the five models are 2,000 points each: a second draw of the data, the four heaviest modes with their weights renormalized, standard deviation 0.3, standard deviation 0.02, and uniform points on \([-2.5,2.5]^2\). Precision is the share of model points within some data point’s distance to its third nearest data point, and recall the share of data points within some model point’s distance to its third nearest model point. The Fréchet distance is computed between Gaussians with the means and covariances of the two samples, with SciPy’s sqrtm. Each panel shows the data in light gray and the first 600 model points in blue, with the three values in the title.

Local figure	Data and construction
gen-eval-precision-recall.png	The eight Gaussians of chapter 6 (circle of radius 2, weights 1/36 to 8/36, standard deviation 0.1), with all draws from one NumPy generator (seed 0). The data are 2,000 points; the five models are 2,000 points each: a second draw of the data, the four heaviest modes with their weights renormalized, standard deviation 0.3, standard deviation 0.02, and uniform points on \([-2.5,2.5]^2\). Precision is the share of model points within some data point’s distance to its third nearest data point, and recall the share of data points within some model point’s distance to its third nearest model point. The Fréchet distance is computed between Gaussians with the means and covariances of the two samples, with SciPy’s `sqrtm`. Each panel shows the data in light gray and the first 600 model points in blue, with the three values in the title.

14. Generative models for science and control {#14-generative-models-for-science-and-control}

Script: ch14_policy.py.

Local figure	Data and construction
gen-science-policy.png	4,000 demonstrations (torch seed 0): lateral positions \(s\) uniform on \([-1,1]\); a side equal to the sign of \(s\), or \(\pm1\) with equal probability where \(\lvert s\rvert<0.3\); and actions equal to the side plus \(\mathcal N(0,0.1^2)\) noise. Squared-error policy: an MLP with two hidden layers of 128 SiLU units from \(s\) to the action, trained with Adam at learning rate \(10^{-3}\) for 3,000 steps of 256 demonstrations drawn with replacement. Diffusion policy: a DDPM on the action with \(T=100\) and \(\beta_t\) evenly spaced from \(10^{-4}\) to 0.2, whose noise predictor maps \((a_t,s,t/T)\) through two hidden layers of 128 SiLU units, trained with Adam at learning rate \(10^{-3}\) for 6,000 steps of 256, and sampled ancestrally, with all its samples drawn in turn from one generator (seed 1). Left: the first 1,500 demonstrations, the squared-error policy at 400 states from −1 to 1, and three diffusion samples at each of them. Other panels: 200 agents starting at evenly spaced lateral positions from −0.2 to 0.2 move forward 0.02 per step for 50 steps and sideways by 0.02 times the action, clipped to \([-1.5,1.5]\), until their forward position reaches 0.9; the obstacle is the gray rectangle of lateral positions from −0.3 to 0.3 and forward positions from 0.9 to 1.1. Every tenth rollout is drawn, for the squared-error policy, the diffusion policy with a new sample at every step, and the diffusion policy with each sample held for 20 steps. The script prints the share of rollouts that enter the obstacle and the share that pass it on the right.

gen-science-policy.png

4,000 demonstrations (torch seed 0): lateral positions \(s\) uniform on \([-1,1]\); a side equal to the sign of \(s\), or \(\pm1\) with equal probability where \(\lvert s\rvert<0.3\); and actions equal to the side plus \(\mathcal N(0,0.1^2)\) noise. Squared-error policy: an MLP with two hidden layers of 128 SiLU units from \(s\) to the action, trained with Adam at learning rate \(10^{-3}\) for 3,000 steps of 256 demonstrations drawn with replacement. Diffusion policy: a DDPM on the action with \(T=100\) and \(\beta_t\) evenly spaced from \(10^{-4}\) to 0.2, whose noise predictor maps \((a_t,s,t/T)\) through two hidden layers of 128 SiLU units, trained with Adam at learning rate \(10^{-3}\) for 6,000 steps of 256, and sampled ancestrally, with all its samples drawn in turn from one generator (seed 1). Left: the first 1,500 demonstrations, the squared-error policy at 400 states from −1 to 1, and three diffusion samples at each of them. Other panels: 200 agents starting at evenly spaced lateral positions from −0.2 to 0.2 move forward 0.02 per step for 50 steps and sideways by 0.02 times the action, clipped to \([-1.5,1.5]\), until their forward position reaches 0.9; the obstacle is the gray rectangle of lateral positions from −0.3 to 0.3 and forward positions from 0.9 to 1.1. Every tenth rollout is drawn, for the squared-error policy, the diffusion policy with a new sample at every step, and the diffusion policy with each sample held for 20 steps. The script prints the share of rollouts that enter the obstacle and the share that pass it on the right.

15. The theory of diffusion models {#15-the-theory-of-diffusion-models}

Script: ch15_memorization.py.

Local figure	Data and construction
gen-theory-memorization.png	Training sets of 32, 8, and 512 points at uniform angles on a circle of radius 2, drawn in that order from one generator (seed 0). The denoiser has the preconditioning of chapter 8 with \(\sigma_{\text{data}}=\sqrt2\) and an MLP with three hidden layers of 128 SiLU units; each network is trained from torch seed 0 for 4,000 steps of 256 training points drawn with replacement, with \(\ln\sigma\sim\mathcal N(-1.2,1.2^2)\) and Adam at learning rate \(10^{-3}\). The exact denoiser of the 32 points is the average of the training points \(x_i\) weighted by the softmax of \(-\lVert x-x_i\rVert^2/(2\sigma^2)\). Sampling: 2,000 points from \(80\,z\) (generator seed 1, the same in each panel) by Heun’s method on 40 levels from 80 to 0.002 with \(\rho=7\), with a final Euler step to 0. Panels: the exact denoiser of 32 points, and networks trained on 8, 32, and 512 points, with the circle in gray and the training points circled in orange. The titles give the share of samples within a tenth of the mean gap \(4\pi/N\) of a training point and the share of 100 equal arcs of the circle that contain a sample.

gen-theory-memorization.png

Training sets of 32, 8, and 512 points at uniform angles on a circle of radius 2, drawn in that order from one generator (seed 0). The denoiser has the preconditioning of chapter 8 with \(\sigma_{\text{data}}=\sqrt2\) and an MLP with three hidden layers of 128 SiLU units; each network is trained from torch seed 0 for 4,000 steps of 256 training points drawn with replacement, with \(\ln\sigma\sim\mathcal N(-1.2,1.2^2)\) and Adam at learning rate \(10^{-3}\). The exact denoiser of the 32 points is the average of the training points \(x_i\) weighted by the softmax of \(-\lVert x-x_i\rVert^2/(2\sigma^2)\). Sampling: 2,000 points from \(80\,z\) (generator seed 1, the same in each panel) by Heun’s method on 40 levels from 80 to 0.002 with \(\rho=7\), with a final Euler step to 0. Panels: the exact denoiser of 32 points, and networks trained on 8, 32, and 512 points, with the circle in gray and the training points circled in orange. The titles give the share of samples within a tenth of the mean gap \(4\pi/N\) of a training point and the share of 100 equal arcs of the circle that contain a sample.

Shared style

genfig.py selects Matplotlib’s Agg backend, and its setup() sets the font DejaVu Sans (10.5 pt, titles 11.5 pt, DejaVu Sans math text), dark axes of line width 0.8 without top and right spines, dark ticks, frameless legends, and white figure, axes, and saved backgrounds. Its save() writes each figure to Sources/Images as a 200 dpi PNG with a tight bounding box and 0.08 inches of padding. The palette is BLUE #2267a5, ORANGE #d46a23, GREEN #39805a, PURPLE #995aa4, RED #b32b35, DARK #152330 for axes and ticks, GRAY #586473, and LIGHT #c1c8cf, with light tints of the first five colors defined for fills; the default color cycle is blue, orange, green, purple, red, and gray. Individual scripts set smaller font sizes for titles, annotations, and legends, and use Matplotlib’s gray_r, gray, Blues, RdBu, and tab10 colormaps for images and classes.