Figure sources
Every figure in the DL chapters is an original plot generated for these notes on 24 September 2026 with PyTorch 2.2.2 (CPU), NumPy, SciPy, Matplotlib, scikit-learn, and NetworkX; no external artwork or course slides are reproduced. Each chapter’s figures come from one script in Sources/Figure code, which is the complete record of the construction. The shared module dlfig.py sets the palette, fonts, and export (opaque white background, 200 dpi PNG). The tables below summarize the data, seeds, and model settings of each figure so that it can be read and regenerated without opening the script; the computing setup gives the environment and commands.
Seeds refer to torch.manual_seed (“torch seed”), torch.Generator().manual_seed (“generator seed”), or numpy.random.default_rng (“NumPy seed”) as stated; scikit-learn’s random_state and NetworkX’s seed are named where they are used, and layers whose initialization is not described keep PyTorch’s default. All scripts call torch.set_num_threads(1). The scripts from chapter 3 on, ch03_optimization.py to ch15_ntk.py, accept figure numbers as command-line arguments (for example python ch09_attention.py 5) and then regenerate only those figures; the numbers follow the order of the figures in the script, which differs from the order in the chapter in chapters 4, 6, and 9, and figure 3 of ch10_self_supervised.py writes both the collapse figure and the masked-autoencoder figure. Without arguments a script regenerates all its figures, and ch01_feedforward.py and ch02_initialization.py always do. According to their docstrings, several scripts take minutes: ch03_optimization.py several minutes, ch08_recurrent.py about eight for figure 3, ch09_attention.py about four for figure 5, ch10_self_supervised.py about ten for figure 3, ch11_scale.py about two for figure 4, ch12_methodology.py about six for figure 3, ch13_graphs.py about one for figure 3, ch14_detection_segmentation.py about six for figures 2 and 3, and ch15_ntk.py about two for figure 1 and ten for figure 2. Training results are single runs unless an average over seeds is stated; they illustrate the claims in the notes and are not benchmarks.
1. Deep feedforward networks {#1-deep-feedforward-networks}
Script: ch01_feedforward.py.
| Local figure | Data and construction |
|---|---|
| dl-activations.png | Sigmoid, tanh, ReLU, leaky ReLU with slope 0.1, GELU, and SiLU from PyTorch on 801 equally spaced points of \([-4,4]\); the derivatives are computed with torch.autograd.grad. |
| dl-universal-approximation.png | Target \(f(x)=\sin2\pi x+x/2\) on \([0,1]\). A one-hidden-layer ReLU network \(c+\sum_j a_j\,\mathrm{ReLU}(x-t_j)\) with \(N\) units is set in closed form, not trained: hinges \(t_j\) at the first \(N\) of \(N+1\) equally spaced knots, \(a_j\) the changes of slope of the linear interpolant, and \(c=f(0)\). Left: \(N=3\), 6, and 24, with the knots as dots. Right: the maximum error on 2,001 grid points for \(N=2,4,\dots,512\), with a reference line of slope \(-2\). |
| dl-sawtooth.png | The tent map \(g(x)=2\,\mathrm{ReLU}(x)-4\,\mathrm{ReLU}(x-1/2)\), two ReLU units that map \([0,1]\) onto itself, composed \(k=1,\dots,4\) times and evaluated on 4,001 points of \([0,1]\); the titles give the \(2^k\) linear pieces and \(2k\) units. |
| dl-linear-regions.png | Untrained ReLU networks on a \(700\times700\) grid of \([-2,2]^2\): one hidden layer of 24 units, three layers of 8, and six layers of 4, each with He-normal weights, biases \(\mathcal N(0,0.5^2)\), and torch seed 3. A region is the set of grid points that share one activation pattern across all layers; regions are colored through a random permutation (NumPy seed 1) cycled over 20 colors, and the titles count the distinct patterns. |
| dl-mlp-spirals.png | Two spirals: angle \(t=3\pi\sqrt u\) with \(u\) uniform on \([0,1]\), a uniform label choosing the sign of \(t(\cos t,\sin t)/3\pi\), and Gaussian noise with standard deviation 0.03; 600 training points (NumPy seed 0) and 4,000 test points (NumPy seed 1). One-hidden-layer ReLU networks with 4, 16, and 64 units and two logits (torch seed 0), trained by full-batch Adam with learning rate 0.01 for 3,000 steps on the cross-entropy. Decision regions on a \(300\times300\) grid of \([-1.15,1.15]^2\) with the training points; the titles give the parameter count and the training and test accuracy. |
2. Initialization and signal propagation {#2-initialization-and-signal-propagation}
Script: ch02_initialization.py.
| Local figure | Data and construction |
|---|---|
| dl-init-scale.png | 512 standard normal inputs through a ReLU network with 50 layers of width 256 and no biases, in float64 (torch seed 0 for each scheme). Four weight initializations: \(\mathcal N(0,0.01^2)\), PyTorch’s default (kaiming_uniform_ with \(a=\sqrt5\), variance \(1/3d\)), Glorot normal (variance \(1/d\) for these square layers), and He normal (variance \(2/d\)). Left: the root mean square of each layer’s activations. Right: a standard normal gradient at the output, sent backward through the same weights and ReLU masks, and its root mean square at each layer. |
| dl-init-tanh.png | 1,000 standard normal inputs in 500 dimensions (torch seed 1) through six tanh layers of width 500 without biases, with weights \(\mathcal N(0,g^2/500)\) for gains \(g=0.5\), 1, and 3; the three networks scale the same Gaussian draws (torch seed 2). Histograms with 80 bins on \([-1,1]\) of the activations at layers 1, 3, and 6; the legends give their standard deviations. |
| dl-init-geometry.png | Left: the infinite-width correlation map of a ReLU layer with He initialization, \(\rho\mapsto(\sqrt{1-\rho^2}+(\pi-\arccos\rho)\rho)/\pi\), iterated over 60 layers from \(\rho_0=0\), 0.5, and 0.9. The dots are the cosine similarity of two inputs of dimension 2,000, constructed with cosine exactly \(\rho_0\), in one bias-free ReLU network of width 2,000 with fresh \(\mathcal N(0,2/2000)\) weights in every layer (torch seed 3). Right: all 128 singular values of the product of 64 random \(128\times128\) matrices, Gaussian with variance \(1/d\) or orthogonal from nn.init.orthogonal_ (torch seed 4). |
3. Optimization for deep networks {#3-optimization-for-deep-networks}
Script: ch03_optimization.py. The data are scikit-learn’s 1,797 handwritten digits (\(8\times8\) pixels divided by 16), the first 1,200 for training and the remaining 597 for testing, and the networks are MLPs with 64 inputs and 10 logits and PyTorch’s default initialization.
| Local figure | Data and construction |
|---|---|
| dl-opt-batch-size.png | One hidden layer of 128 ReLU units (torch seed 0), SGD with momentum 0.9 on minibatches drawn with replacement (generator seed 0), for batch sizes \(B=4,8,\dots,1024\) and 13 learning rates log-spaced from \(10^{-2.5}\) to \(10^{0.5}\). A run counts the steps until the cross-entropy on all 1,200 training digits, checked every 5 steps, falls below 0.05; it fails if that loss exceeds 50 or is not finite, or after 6,000 steps. Each \(B\) keeps its fastest learning rate, and \(S(B)=S_{\min}(1+B_{\text{noise}}/B)\) is fitted by least squares on the log scale, with \(B_{\text{noise}}\) searched over 400 log-spaced values from 1 to \(10^4\). Panels: steps, examples (steps times \(B\)), and the best learning rate. |
| dl-opt-schedules.png | 20% of the training labels are replaced by labels drawn uniformly from the ten classes (generator seed 5). One hidden layer of 64 ReLU units (torch seed 0), SGD with momentum 0.9, batch size 32 (37 steps per epoch, shuffling generator seed 1), 40 epochs, peak learning rate 0.3. Schedules: constant; step decay by a factor of 10 at 50% and at 75% of the steps; cosine; linear warmup over 5% of the steps followed by cosine; and the same warmup, a constant rate, and linear decay to zero over the last 20%. Left: the schedules, with clean test accuracy in the legend; right: the training loss on the noisy labels after each epoch. |
| dl-opt-edge-of-stability.png | The first 500 digits with one-hot targets and loss \(\frac12\lVert f(x)-y\rVert^2\) averaged over examples. A tanh MLP with two hidden layers of 64 units (torch seed 0), trained by full-batch gradient descent for 6,000 steps at \(\eta=0.2\), 0.1, and 0.05. Left: the sharpness \(\lambda_{\max}\) of the Hessian every 50 steps, estimated by 20 power iterations with Hessian–vector products from autograd, and dashed lines at \(2/\eta\). Right: the loss at every step. |
| dl-opt-connectivity.png | Two networks A and B with one hidden layer of 512 ReLU units (torch and generator seeds 1 and 2), each trained for 40 epochs with SGD, learning rate 0.05, momentum 0.9, and batch size 32. Training cross-entropy at 26 equally spaced points of the segment \((1-\alpha)\theta_0+\alpha\theta_1\) from A’s initialization to A, from A to B, and from A to B after B’s hidden units are permuted to match A’s. The permutation maximizes the summed correlation of standardized hidden activations on the training set (scipy.optimize.linear_sum_assignment). |
4. Normalization and residual connections {#4-normalization-and-residual-connections}
Script: ch04_normalization_residual.py. The trained networks (torch seed 0) map the 64 pixels of a digit to width 64 with a He-normal linear layer and ReLU, apply blocks of two linear layers of width 64 with He-normal weights, zero biases, and optional normalization after each, where a plain block returns \(\mathrm{ReLU}(h)\) and a residual block \(\mathrm{ReLU}(x+h)\) for branch output \(h\), and end in a linear output layer; training uses the first 1,200 digits (pixels divided by 16) and SGD with momentum 0.9, batch size 64, and 15 epochs (shuffling generator seed 1), and records the mean minibatch loss of each epoch.
| Local figure | Data and construction |
|---|---|
| dl-norm-learning-rate.png | Plain networks of four blocks, nine linear layers of width 64 before the output layer, without normalization or with BatchNorm1d after every linear layer in the blocks, trained at learning rates 0.003, 0.01, 0.03, 0.1, 0.3, and 1. Left: the loss of the last epoch, with runs whose loss is not finite drawn on the “diverged” line. Right: the loss per epoch at learning rate 0.1. |
| dl-norm-axes.png | A schematic, not computed: a batch of six examples with eight channels, with the cells that share one mean and variance highlighted for batch norm (one channel), layer norm (one example), instance norm (one example and one channel), and group norm (one example and a group of four channels). |
| dl-residual-depth.png | Networks of 4 and 24 blocks (10 and 50 linear layers) trained at learning rate 0.01 in three variants: plain with batch normalization, residual with batch normalization, and residual without normalization whose second linear layer in each branch starts with zero weights. Mean training loss per epoch on a log scale. |
| dl-residual-signal.png | No training: 256 standard normal inputs of width 512 (torch seed 0) through 64 blocks with fresh bias-free He-normal weights \(\mathcal N(0,2/512)\) in every layer. Left: the root mean square of the stream after each block for a plain ReLU layer and for a residual branch \(b=W_2\,\mathrm{ReLU}(W_1x)\) added as \(b/\sqrt2\), as \(b\) standardized per coordinate over the 256 inputs, or as \(b/\sqrt{2L}\) with \(L=64\). Right: the cosine similarity of two orthogonal inputs of equal norm in the plain ReLU network and in the residual network with branches \(b/\sqrt{2L}\) (weights from torch seed 1). |
5. Regularization and generalization in deep networks {#5-regularization-and-generalization-in-deep-networks}
Script: ch05_regularization.py. Networks are ReLU MLPs on the digits (pixels divided by 16) with PyTorch’s default initialization, and randomized labels are drawn uniformly from the ten classes, so some coincide with the true ones.
| Local figure | Data and construction |
|---|---|
| dl-reg-memorization.png | The first 1,200 digits for training, with 0, 20, 50, or 100% of their labels randomized (generator seed 3, which also shuffles), and the remaining 597 with clean labels for testing. Two hidden layers of 512 units (torch seed 0), Adam with learning rate \(10^{-3}\), batch size 64, 150 epochs. Accuracy on the noisy training labels and on the test set after every epoch. |
| dl-reg-comparison.png | The first 600 digits for training, the next 300 for validation, and the remaining 897 for testing, with 20% of the training and validation labels randomized (generator seed 9). Two hidden layers of 512 units, AdamW with learning rate \(10^{-3}\) and no weight decay unless stated, batch size 32, 100 epochs. Methods: label smoothing 0.1; dropout 0.5 after each hidden layer; mixup with \(\lambda\sim\mathrm{Beta}(1,1)\) per batch; weight decay 3; early stopping at the epoch of best validation accuracy; an ensemble averaging the probabilities of five networks; and dropout with weight decay and early stopping. Bars show the mean and standard deviation of clean test accuracy over seeds 0, 1, and 2 (torch, generator, and NumPy), the ensembles using seeds \(10s+k\) for \(k=0,\dots,4\). |
| dl-reg-averaging.png | The first 600 digits with 20% of their labels randomized (generator seed 9) for training, and the remaining 1,197 with clean labels for testing. Eight networks with one hidden layer of 256 units (torch and generator seeds 0–7), SGD with learning rate 0.05, momentum 0.9, batch size 32, 60 epochs. Curves: test accuracy of the ensemble that averages the probabilities of the first \(k\) networks and of the single network whose weights average theirs. Horizontal lines: mean accuracy of the final networks and of each run’s weights averaged over the ends of epochs 31–60. |
6. Convolutional networks {#6-convolutional-networks}
Script: ch06_convolution.py.
| Local figure | Data and construction |
|---|---|
| dl-conv-operation.png | A \(6\times6\) input of integers from 0 to 3 (NumPy seed 3) and the kernel \(\begin{psmallmatrix}1&0&-1\\2&0&-2\\1&0&-1\end{psmallmatrix}\). The \(4\times4\) output is the valid cross-correlation; the output in row 1 and column 2 (counting from 0) and its input patch are highlighted. |
| dl-conv-feature-maps.png | scikit-learn’s sample image china.jpg, averaged over the color channels, subsampled by 2 in each direction, and scaled to \([0,1]\). It is convolved with zero padding by the vertical and horizontal Sobel kernels, a \(5\times5\) average, and the Laplacian \(\begin{psmallmatrix}0&1&0\\1&-4&1\\0&1&0\end{psmallmatrix}\). Signed responses are shown in a red–blue map clipped at the 99th percentile of their absolute values, the blur in gray. |
| dl-conv-shift.png | The digits (pixels divided by 16) zero-padded to \(14\times14\), the first 1,200 for training and the remaining 597 for testing. An MLP with one hidden layer of 128 units and a CNN with \(3\times3\) convolutions of 32, 32, and 64 channels, a \(2\times2\) max pooling after the second, global max pooling, and a linear output (both torch seed 0), trained on unshifted images with Adam, learning rate \(3\times10^{-3}\), batch size 64, 40 epochs (shuffling generator seed 0). Test accuracy after shifting every test image by \((dx,dy)\in\{-3,\dots,3\}^2\) with torch.roll, which at these shifts wraps only padding around; the titles give the parameter counts. |
| dl-conv-receptive-field.png | Untrained networks of \(3\times3\) convolutions with 16 channels, padding 1, He-normal weights, and ReLU: 2, 5, and 10 layers with stride 1, and 5 layers with stride 2 in layers 2 and 4. For each of torch seeds 0–19, a network and a standard normal \(65\times65\) input give the absolute gradient of the central output, summed over channels, with respect to every input pixel; the maps are summed over seeds and scaled to a maximum of 1. The green squares are the theoretical receptive fields from the recursion over kernel size and stride. |
7. Convolutional architectures and transfer learning {#7-convolutional-architectures-and-transfer-learning}
Script: ch07_architectures_transfer.py. Neither figure trains a model.
| Local figure | Data and construction |
|---|---|
| dl-arch-accuracy-compute.png | A table of 18 ImageNet classifiers from AlexNet (2012) to ConvNeXt-B (2022), hard-coded from the torchvision 0.29 model table, with year, top-1 accuracy, parameters, and GFLOPs per \(224\times224\) image. Accuracy against GFLOPs on a log scale, colored by year, with marker area growing with the parameter count. The red arrow marks ResNet-50 at 76.13% and at 80.858% with a modern training recipe. |
| dl-arch-blocks.png | A schematic drawn with Matplotlib patches: a VGG stage (two \(3\times3\) convolutions with 256 channels and ReLU, \(2\times2\) max pooling); a ResNet bottleneck (\(1\times1\) to 64, \(3\times3\) with 64, \(1\times1\) to 256, batch norm, identity shortcut); a MobileNetV2 inverted residual (\(1\times1\) to 144, \(3\times3\) depthwise, ReLU6, \(1\times1\) to 24 without activation); and a ConvNeXt block (\(7\times7\) depthwise with 96 channels, LayerNorm, \(1\times1\) to 384, GELU, \(1\times1\) back to 96). |
8. Recurrent networks {#8-recurrent-networks}
Script: ch08_recurrent.py.
| Local figure | Data and construction |
|---|---|
| dl-rnn-unrolled.png | A schematic drawn with Matplotlib patches: the folded recurrence \(x_t\to h_t\to\hat y_t\) with weights \(W_{xh}\), \(W_{hh}\), and \(W_{hy}\), and the same network unrolled from \(h_0\) over steps \(1,\dots,T\). |
| dl-rnn-gradients.png | 16 sequences of 120 inputs \(x_t\sim\mathcal N(0,0.5^2I)\) in 64 dimensions (torch seed 1), added directly to the recurrence from \(h_0=0\), with \(W_{hh}\) equal to \(g\) times an orthogonal matrix (torch seed 0). For a random unit vector \(v\) per sequence, autograd gives the gradient of \(v^\top h_T\) with respect to every earlier state, and the curves show its norm, averaged over sequences, against the lag \(k\). Left: \(h_t=W_{hh}h_{t-1}+x_t\) with \(g=0.9\), 1.0, and 1.1. Right: \(h_t=\tanh(W_{hh}h_{t-1}+x_t)\) with \(g=1\), 1.5, and 3, and an LSTMCell with 64 units, default weights, and zero biases except a forget-gate bias of 3, measured on the cell state. |
| dl-rnn-adding.png | The adding problem: sequences of length 100 with a channel of values uniform on \([0,1]\) and a marker channel with one mark in each half, the target being the sum of the two marked values; a fresh batch of 64 sequences at every step (generator seed 1). A tanh RNN, a GRU, and two LSTMs with 64 hidden units and a linear readout of the last state (torch seed 0); one LSTM keeps PyTorch’s default initialization and the other has forget-gate bias 1, split equally between its two bias vectors. Adam with learning rate \(10^{-3}\), gradient clipping at norm 1, squared error, 4,000 steps. The curves are 50-step moving averages; the dashed line at \(1/6\) is the error of predicting the mean. |
9. Attention and transformers {#9-attention-and-transformers}
Script: ch09_attention.py.
| Local figure | Data and construction |
|---|---|
| dl-attn-lookup.png | 40 keys at sorted angles uniform on \([-\pi,\pi]\), stored as unit vectors on the circle, with values \(\sin2\theta+\frac12\cos\theta\) plus Gaussian noise with standard deviation 0.25 (NumPy seed 0). A query of norm 1, 10, or 1,000 in direction \(\theta\) gives the softmax of its dot products with the keys as weights. Left: the weights for \(\theta=0.8\). Right: the weighted average of the values for 600 query angles in \([-\pi,\pi]\). |
| dl-attn-matrices.png | Six tokens with standard normal embeddings \(X\in\mathbb R^{6\times8}\) and \(W_Q,W_K,W_V\in\mathbb R^{8\times4}\) with entries \(1.6\,\mathcal N(0,1)/\sqrt8\) (torch seed 3). The panels show \(Q\), \(K^\top\), \(S=QK^\top/\sqrt{d_k}\), the row-wise softmax \(A\) of \(S\) with the entries above the diagonal set to \(-\infty\), \(V\), and \(Y=AV\). The words “the cat sat on the mat” are only labels. |
| dl-attn-positions.png | Sinusoidal encodings \(p_{t,2j}=\sin\omega_jt\) and \(p_{t,2j+1}=\cos\omega_jt\) with \(\omega_j=10000^{-2j/d}\). Left: the sine coordinates for positions 0 to 127 with \(d=64\). Right: \(p_{100}\cdot p_{100+k}/(d/2)\) for offsets \(k=0,\dots,999\) with \(d=64\) and 256, on an axis that is linear up to 10 and logarithmic beyond. |
| dl-attn-block.png | A schematic drawn with Matplotlib patches. Left: a pre-normalization block (LayerNorm, multi-head attention, and a residual addition; LayerNorm, an MLP \(d\to4d\to d\), and a residual addition) repeated \(L\) times. Right: the allowed query–key pairs for full attention on six tokens (encoder only), causal attention (decoder only), and an encoder–decoder with \(6\times6\) encoder self-attention, \(5\times5\) causal decoder self-attention, and \(5\times6\) cross-attention. |
| dl-attn-adding.png | The adding problem of chapter 8 with the same batches (generator seed 1), Adam with learning rate \(10^{-3}\), gradient clipping at norm 1, and 1,500 steps, for a tanh RNN and a GRU with 64 units and a transformer, all from torch seed 0. The transformer embeds each step linearly to width 64, prepends a learned readout token, adds learned position embeddings (both initialized as \(\mathcal N(0,0.02^2)\)), and applies one pre-normalization block with 4-head nn.MultiheadAttention and a GELU MLP of width 256, then LayerNorm and a linear readout of the readout token. Left: 50-step moving averages of the squared error. Right: each head’s attention weights from the readout token to the 100 positions of one test sequence (generator seed 123), with its target and prediction in the title. |
| dl-attn-vit.png | scikit-learn’s china.jpg cropped to the \(427\times427\) square of columns 100 to 526, resized to \(224\times224\) by area interpolation, and cut into \(7\times7=49\) patches of \(32\times32\) pixels. Right: the sequence of 50 tokens, a class-token placeholder followed by the patches in raster order. No model is run. |
10. Self-supervised representation learning {#10-self-supervised-representation-learning}
Script: ch10_self_supervised.py. The digits (pixels divided by 16) are split 60/40 into 1,078 training and 719 test images, stratified by class (train_test_split, random_state=0), and a view of an image is a random affine transform (rotation up to 0.3 radians, scaling by up to 15%, shift up to one pixel, bilinear sampling with zero padding) plus Gaussian pixel noise with standard deviation 0.1.
| Local figure | Data and construction |
|---|---|
| dl-ssl-families.png | A schematic drawn with Matplotlib patches of the contrastive, self-distillation, and masked-modeling objectives. The masked-modeling panel shows the fourth test image with 8 of its 16 patches of \(2\times2\) pixels hidden (NumPy seed 0). |
| dl-ssl-views.png | The first training images of classes 2, 5, and 7, each with six random views (generator seed 4) clipped to \([0,1]\). |
| dl-ssl-collapse.png | Encoder \(f\): an MLP \(64\to512\to256\to128\) with ReLU; projector: ReLU and a linear map to 64, or for SimSiam two linear layers with batch normalization and a predictor \(64\to32\to64\). Each method starts from torch seed 0 and trains with Adam, learning rate \(10^{-3}\), for 3,000 steps on two views of 256 training images drawn with replacement (generator seed 1); the losses are the negative cosine similarity of the two projections (positive pairs only), InfoNCE with temperature 0.2 (SimCLR), VICReg with weights 25, 25, and 1, and the symmetric SimSiam loss with and without stop-gradient. Left: the effective rank, the exponential of the entropy of the normalized singular values, of the centered 128-dimensional representation of all training images every 50 steps. Right: test accuracy of standardized logistic regression (max_iter=5000) fitted on \(k=1,2,5,10,20,50\) training images per class, averaged over NumPy seeds 0–9, for these encoders, raw pixels, the encoder at initialization, and the masked autoencoder of the next row, whose features are its encoder outputs averaged over the 16 patches. |
| dl-ssl-mae.png | A masked autoencoder on the 16 patches of \(2\times2\) pixels: a linear patch embedding to width 64 with learned positions, an encoder of three pre-normalization transformer layers (4 heads, MLP width 256, GELU, no dropout) on 8 randomly kept patches, and a one-layer decoder of the same kind with mask tokens, trained on the squared error of the hidden patches. AdamW with learning rate \(10^{-3}\) and weight decay 0.05, 4,000 steps of 128 training images (torch seed 0, generator seed 1 for batches and masks). Shown: the first ten test images, their visible patches under a new random mask (generator seed 7), and the reconstruction, which keeps the visible pixels. |
11. Training at scale and efficient inference {#11-training-at-scale-and-efficient-inference}
Script: ch11_scale.py.
| Local figure | Data and construction |
|---|---|
| dl-scale-roofline.png | Analytic: attainable throughput \(\min(\text{peak},\ \text{bandwidth}\times\text{intensity})\) for an A100 SXM (312 TFLOP/s bf16, 2.039 TB/s) and an H100 SXM (989 TFLOP/s bf16, 3.35 TB/s), with the ridge points marked. Operations are placed on the H100 roof at their intensity with 2-byte numbers: adding two tensors (1/6 FLOP per byte), LayerNorm, softmax, and GELU (1.5), decoding one token (1), decoding a batch of 64 through a \(4096\times4096\) matrix (\(64\cdot4096/(4096+128)\)), and \(n\times n\times n\) matrix products (\(n/3\)) for \(n=256\) and 4,096. |
| dl-scale-zero.png | Analytic: memory per GPU for \(\Psi=7.5\times10^9\) parameters on 64 GPUs, counting \(2\Psi\) bytes of 16-bit parameters, \(2\Psi\) of 16-bit gradients, and \(12\Psi\) of 32-bit master weights, momentum, and variance, with each sharded part divided by 64. The bars shard nothing, the optimizer state (ZeRO-1), also the gradients (ZeRO-2), and also the parameters (ZeRO-3, FSDP), for totals of 120, 31.4, 16.6, and 1.9 GB; activations are not counted. |
| dl-scale-pipeline.png | A simulated timeline for 4 stages and 8 micro-batches, with forward passes of 1 time unit and backward passes of 2. GPipe runs all forward passes and then all backward passes; 1F1B gives stage \(s\) (counting from 0) \(4-s\) warm-up forward passes and then alternates backward and forward, each pass starting as soon as its stage is free and the pass it depends on has finished. The titles report the idle fraction, one minus busy time over 4 times the total time, and the largest number of micro-batches whose activations a stage holds. |
| dl-scale-compression.png | The digits (pixels divided by 16) split 60/40 into 1,078 training and 719 test images, stratified by class (random_state=0). An MLP \(64\to256\to256\to10\) with ReLU (torch seed 0), trained with Adam, learning rate \(10^{-3}\), for 2,000 steps of 64 examples drawn with replacement (generator seed 1). Left: the weights, not the biases, rounded symmetrically to \(b=8\), 6, 5, 4, 3, and 2 bits with integer levels up to \(2^{b-1}-1\) and one scale per matrix or per output unit, set by the largest absolute weight. Right: the 50, 70, 80, 90, 95, or 98% smallest-magnitude weights of each layer set to zero, without retraining, then with 300 steps of Adam with the zeros kept (generator seed 1), and gradually, pruning to each sparsity in turn with 300 fine-tuning steps after each (generator seeds 10–15). |
12. Practical methodology {#12-practical-methodology}
Script: ch12_methodology.py. The digits (pixels divided by 16) are split 60/40 into 1,078 training and 719 validation images, stratified by class (random_state=0), and unless stated the model is an MLP with one hidden layer of 128 ReLU units (torch seed 0) trained on minibatches of 64 drawn with replacement (generator seed 1).
| Local figure | Data and construction |
|---|---|
| dl-method-curves.png | Left: Adam at learning rates \(10^{-4}\), \(3\times10^{-3}\), 0.1, and 1 for 1,500 steps; the minibatch losses, capped at 50, are smoothed by a 25-step moving average. Right: 100 training images (NumPy seed 0) and an MLP with two hidden layers of 512 units (torch seed 0), trained with Adam, learning rate \(10^{-3}\), on batches of 32 of these images for 3,000 steps. The training loss on the 100 images and the validation loss and accuracy are recorded every 25 steps. |
| dl-method-lr-range.png | A range test: 300 Adam steps with the learning rate increased exponentially from \(10^{-6}\) to 10, the loss smoothed by a bias-corrected exponential moving average with factor 0.9. The curves stop 20 steps after the smoothed loss first exceeds four times its minimum over the first 250 steps; the dotted line marks the learning rate of the lowest smoothed loss before that point and the dashed line a tenth of it. Orange: validation accuracy of full runs of 1,000 Adam steps at 11 fixed learning rates log-spaced from \(10^{-5}\) to 1. |
| dl-method-search.png | Left: validation accuracy of 625 AdamW runs of 400 steps on a \(25\times25\) grid of \(\log_{10}\) learning rate and \(\log_{10}\) weight decay, each from \(-5\) to 0, with 9 grid trials at the centers of the thirds of each range and 9 random trials uniform on \([-5,0]^2\) (NumPy seed 3), all trained. Right: the expected best of \(k\) random trials, computed exactly for draws with replacement from the 625 accuracies, and the average best of a \(k\times k\) grid for \(k=1,\dots,5\) over 2,000 random offsets (NumPy seed 4), with grid points snapped to the nearest run. The purple curve places the same grid values at \(k^4\) trials, as if two further hyperparameters had no effect. |
13. Graph neural networks {#13-graph-neural-networks}
Script: ch13_graphs.py. Message passing uses \(\hat A=D^{-1/2}(A+I)D^{-1/2}\), with \(D\) the degree matrix of \(A+I\).
| Local figure | Data and construction |
|---|---|
| dl-gnn-karate.png | NetworkX’s karate club graph, labeled by each member’s club, with one-hot node features. Left: the graph in a spring layout (seed 3). Middle: the output of a GCN with widths \(34\to16\to8\to2\), each layer \(\hat A(HW+b)\) with tanh between layers, at initialization (torch seed 0). Right: the same architecture from the same seed with a linear head \(2\to2\) on the tanh of its output, trained with Adam, learning rate 0.01, for 300 full-graph steps on the cross-entropy of nodes 0 and 33 only; misclassified nodes are annotated. |
| dl-gnn-wl.png | 1-WL color refinement, four rounds from a uniform coloring, on a 6-cycle and two disjoint triangles, and on the ten-carbon skeletons of decalin and bicyclopentyl written as edge lists; nodes are colored by their final color. The first pair is drawn on circles, the molecules with the Kamada–Kawai layout. |
| dl-gnn-oversmoothing.png | A stochastic block model with four blocks of 100 nodes and edge probabilities 0.1 within and 0.01 between blocks (NetworkX seed 0). Node features are a class mean \(0.5\,\mathcal N(0,I_{16})\) plus standard normal noise, and 80 random nodes are labeled for training, the other 320 used for testing (all generator seed 0). Left: for \(k=0,\dots,64\), the mean of the cosine-similarity matrix of the rows of \(\hat A^kX\) and the mean distance between the four class centroids of the row-normalized features, relative to \(k=0\). Right: a linear layer \(16\to32\) with ReLU, \(L\) layers \(h\mapsto\mathrm{ReLU}(\hat A(hW+b))\) with or without residual addition, and a linear output, for \(L=0,1,2,4,8,16,32\); Adam with learning rate 0.01 and weight decay \(5\times10^{-4}\) for 200 full-graph steps, test accuracy averaged over torch seeds 0–2. |
14. Detection and segmentation {#14-detection-and-segmentation}
Script: ch14_detection_segmentation.py. A scene is a \(64\times64\) RGB image with a background of level 0.35, Gaussian noise with standard deviation 0.08, and a random sinusoidal pattern of amplitude 0.1, on which one to three circles, squares, or triangles of random class, size parameter uniform on \([6,13]\) pixels, position, and color are painted in turn, the boxes and masks covering the visible parts and objects with fewer than 12 visible pixels dropped; the training sets have 2,000 scenes (NumPy seed 0) and the test sets 200 (NumPy seed 1).
| Local figure | Data and construction |
|---|---|
| dl-det-tasks.png | One scene (NumPy seed 11), redrawn until it has three objects of exactly two classes, shown as the image, its boxes and classes, its class mask, and its instance mask. |
| dl-det-detector.png | A center-based detector: six \(3\times3\) convolutions with batch norm and ReLU (16, 32, 32, 64, 64, and 64 channels, stride 2 in the second and fourth) giving a \(16\times16\) map, and \(1\times1\) heads for three class heatmaps (bias initialized to \(-2.19\)) and four box outputs (width, height, and sub-cell offset). The targets are Gaussian bumps of width \(\max(0.5,(w+h)/12)\) cells at each object’s center cell, and the loss is a focal loss with \(\alpha=2\) and \(\beta=4\) on the heatmaps plus 0.5 times the \(L_1\) error of the box outputs at centers. Adam with learning rate \(2\times10^{-3}\), 1,500 steps of 16 scenes drawn with replacement (torch seed 0, generator seed 1). Detections are the local maxima in \(3\times3\) windows above 0.05, at most 10 per scene, and AP is the 101-point interpolated average precision per class, averaged over the classes; the figure shows the first four test scenes detected without error at score 0.3 and the first scene with an error among those with at most two objects. |
| dl-det-unet.png | A U-Net with blocks of two \(3\times3\) convolutions with batch norm and ReLU (8, 16, 32, and 64 channels), three \(2\times2\) max poolings to \(8\times8\), \(2\times2\) transposed convolutions back up with concatenated skips, and a \(1\times1\) output for the background and the three shapes; the variant without skips omits the concatenations. Each is trained from torch seed 0 with Adam, learning rate \(2\times10^{-3}\), for 800 steps of 16 scenes (generator seed 1) on the pixelwise cross-entropy. The IoU and boundary accuracy in the caption are computed on all 200 test scenes, a boundary pixel being one within one pixel, diagonals included, of a class change in the ground truth. Shown: test scenes 0, 3, 5, 6, and 8 with the ground truth and both predictions. |
15. Infinite width and the neural tangent kernel {#15-infinite-width-and-the-neural-tangent-kernel}
Script: ch15_ntk.py. Computations are in float64 on inputs \((\cos\theta,\sin\theta)\), with one-hidden-layer networks \(f(x)=a^\top\mathrm{ReLU}(Wx+b)/\sqrt n\) in the NTK parameterization, \(W\) and \(a\) standard normal and \(b\sim\mathcal N(0,1/4)\) (generator seed 1), trained by full-batch gradient descent with learning rate 1 on half the mean squared error.
| Local figure | Data and construction |
|---|---|
| dl-ntk-lazy.png | The target \(\sin\theta+\frac12\sin3\theta\) at 20 angles uniform on \([0,2\pi)\) (generator seed 0), with 300 equally spaced test angles; widths 64, 256, 1,024, 4,096, and 16,384, trained for 4,000 steps. Kernel regression uses the empirical NTK at initialization, \(f_0(x)+\Theta_0(x,X)\Theta_0(X,X)^{-1}(y-f_0(X))\), the limit of training the linearized network. Left: the relative Frobenius change of the NTK on the training inputs, the relative change of the parameter vector, and the relative distance between the trained network and the kernel prediction on the test angles, with a reference line of slope \(-1/2\). Middle and right: both predictions for 64 and 16,384 units. |
| dl-ntk-spectral-bias.png | 256 equally spaced angles and a network with 4,096 units. Left: for \(k=1,\dots,40\), the Rayleigh quotient of \(\sin k\theta\) under the empirical NTK matrix \(J_0J_0^\top/256\) at initialization. Right: 30,000 steps on the target \(\sin\theta+\sin4\theta+\sin12\theta\), with the residual’s amplitude \(\lvert\frac2N\sum_i r_i\sin k\theta_i\rvert\) at \(k=1\), 4, and 12 recorded at log-spaced steps (gradient descent with learning rate 1 on half the mean squared error; the network is initialized with generator seed 1). The dashed curves apply the same updates to the linearized network, \(r_t=(I-\eta K)^tr_0\) with the initial NTK matrix \(K\), computed exactly from its eigendecomposition; the script also prints the change of the kernel matrix after training. |
Shared style
dlfig.py selects Matplotlib’s Agg backend, and its setup() sets the font DejaVu Sans (10.5 pt, titles 11.5 pt, DejaVu Sans math text), dark axes without top and right spines, frameless legends, and white figure, axes, and saved backgrounds. Its save() writes each figure to Sources/Images as a 200 dpi PNG with a tight bounding box. The palette is BLUE #2267a5, ORANGE #d46a23, GREEN #39805a, PURPLE #995aa4, RED #b32b35, DARK #152330 for axes and ticks, GRAY #586473, and LIGHT #c1c8cf, with light tints of the first five colors for fills; the default color cycle is blue, orange, green, purple, red, and gray.