Figure sources
Sources and construction details for figures in the Foundations notes. The original terminology diagrams were checked on 20 September 2026; the illustrated examples below were added on 23 September 2026. Four further figures for chapter 1 were added on 25 September 2026. On 26 September 2026 chapter 3 received twenty-two drawn and ten imported figures and its two optimizer-path figures were redrawn, chapter 2 received one drawn and one imported figure, and chapter 4 received twenty-nine drawn figures and had its seven earlier figures redrawn. On 27 September 2026 chapter 5 received nine drawn figures and had its four earlier figures redrawn, and chapter 6 received six drawn figures. Unlike the earlier authored plots, the figures added from 25 September onward are regenerated by bundled code in Sources/Figure code (see Reproducing the figures). Imported images are retained without modification, except for the captured Distill figures and the figure cropped from a PDF, whose changes are listed with them. Newly plotted figures have opaque white backgrounds. The image files themselves are unchanged.
| Local image | Creator | Original source | License | Use in the note |
|---|---|---|---|---|
| Colored neural network.svg | Glosser.ca | Colored neural network.svg | CC BY-SA 3.0 | Labeled, color-coded input, hidden, and output layers; original SVG used without modification |
| Clustering.svg | Hellisp; vector version by Wgabrie | Cluster-2.svg | Public domain, as stated on the source page | A possible grouping of observations by similarity |
| Linear regression.svg | Sewaqu; transparent-background revision by RaphaelQS | Linear regression.svg | Public domain, as stated on the source page | Earlier generic linear-model illustration; retained in Sources and replaced inline by the apartment example below |
| Reinforcement_learning_diagram.svg.png | Megajuice | Reinforcement learning diagram.svg | CC0 1.0 | Agent–environment feedback loop; reuses the PNG already stored in this folder |
The learning–prediction–action diagram is authored directly in the note as editable Mermaid.
Terminology: illustrated examples
These figures accompany Terminology and Mathematical Language. The selected photograph shows Kasparov on the left and the computer terminal on the right during the 1996 match. Earlier candidate photographs remain in Sources, with their credits below; all downloaded photographs are retained without modification. The remaining figures are original plots generated for these notes using NumPy, SciPy, Matplotlib, and scikit-learn; they are not copied course slides. On 25 September 2026 the depth-two iris tree, whose two questions both used petal width, was replaced by a tree that uses both petal measurements; the earlier tree and its region plot remain in Sources. The last four rows are produced by Sources/Figure code/ch01_terminology.py, which also prints every number the chapter quotes from them.
| Local image | Source, attribution, and construction |
|---|---|
| Kasparov and the computer terminal, 1996 | Selected figure. Kasparov at left, computer terminal at right, during the 1996 match in Philadelphia. Reproduced unchanged from RTVE’s account of the first Deep Blue match, published 10 February 2021; source image filename 1612969862481.jpg. The retrieved page does not identify the photographer for this image, so no photographer credit is inferred. Copyrighted source image included in these private study notes; no open license is claimed. The caption identifies the event without claiming that the photograph captures the exact moment of resignation. |
| Kasparov at the board, 1997 | © Stan Honda/AFP. Earlier candidate, no longer embedded. Direct photograph of Kasparov with his head in his hands during the 1997 Deep Blue rematch. Downloaded unchanged from the header image of RTBF’s historical article, and compared with the Getty Images agency record, image 51654393, which identifies the photographer and dates the photograph to 10 May 1997, during game five. It is not a photograph of the moment of defeat. Copyrighted editorial photograph, included in these private study notes; no open license is claimed. |
| Deep Blue and Kasparov, 1997 | Photograph by Najlah Feanny, © Najlah Feanny/CORBIS SABA. Computer History Museum, “Garry Kasparov vs. Deep Blue”, accession NF1108205, 1997. Earlier figure, retained in Sources but no longer embedded: spectator monitors show the board and Kasparov. The museum page does not state an open license; this copyrighted photograph is included in these private study notes and is not relicensed with the original plots. |
| Learned decision tree | Earlier figure, retained in Sources but no longer embedded. Original drawing of a fitted DecisionTreeClassifier(max_depth=2, random_state=7). Trained on all 150 Iris observations, using petal length and width; both selected splits use petal width. Thresholds, leaf labels, and branches are taken from the fitted estimator. The fit makes six errors on its training observations. |
| Classification regions | Earlier figure, retained in Sources but no longer embedded. The same depth-two Iris tree. Point colors and shapes denote observed species; region colors denote predictions. Observations displayed here were used for fitting, not a held-out evaluation. |
| Regression example | Original synthetic data: 28 evenly spaced areas from 30 to 120 square meters, rent = 500 + 18 × area + Normal(0, 190²), NumPy seed 14. Ordinary least squares with an intercept gives about 1,945 dollars/month at area 80. These are illustrative prices. |
| PCA representation | Original synthetic correlated measurements, 55 observations, NumPy seed 23. Centered SVD gives the first principal direction; its sign is chosen to make its first component positive. The right panel shows the projection coordinate. Orange marks the same observation in both panels; the orange cross on the left marks its projection. |
| Deep digit recognizer | Actual activations and class probabilities from an MLPClassifier with ReLU hidden layers of sizes 64, 32, and 16, Adam updates, and seed 7. Inputs are the 64 optical-digit measurements divided by 16. A stratified 75%/25% split of scikit-learn’s 1,797 observations is used for this illustration; the displayed digit was excluded from training. Hidden vectors are reshaped only for display, with independent color scales per layer. This is a fully connected network, not LeNet or a convolutional model. |
| Decision tree using both measurements | Embedded, added 25 September 2026. Drawing of a fitted DecisionTreeClassifier(max_leaf_nodes=4, random_state=2) on all 150 Iris observations, using petal length and width. Best-first growth gives petal length ≤ 2.45 cm, then petal width ≤ 1.75 cm, then petal length ≤ 4.95 cm. The first split ties with petal width ≤ 0.80 cm, which also isolates Setosa; this random_state selects the length split. Thresholds, leaf labels, and branches are taken from the fitted estimator. Four training errors. |
| Classification regions, two measurements | Embedded, added 25 September 2026. The same four-leaf tree evaluated on a 700 × 400 grid of the petal plane. Point colors and shapes denote observed species; region colors denote predictions. One misclassified Versicolor at (4.8, 1.8) cm is hidden beneath two Virginica observations with identical measurements. Observations displayed here were used for fitting, not a held-out evaluation. |
| Clear and ambiguous digits | Added 25 September 2026. An MLPClassifier with the architecture of the deep digit recognizer (ReLU hidden layers of 64, 32, and 16 units, Adam, random_state=7, max_iter=2000), trained on a stratified 75%/25% split with random_state=7 of the inputs divided by 16. Dataset indices 1792, 1038, and 1765 lie in the held-out quarter. Predicted probabilities: 1.00 for 9; 0.641 for 9 and 0.358 for 3; 0.741 for 3 and 0.257 for 5. Labels are the recorded dataset labels. This network is trained separately from the one in the deep digit recognizer figure. The same split gives the held-out accuracies quoted in the chapter’s Appendix A: network 98.0%, 1-nearest neighbor 98.9%, logistic regression 97.8%, and k-means with 10 or 50 clusters named by their majority training label 78.4% or 95.3% (n_init=10, random_state=0). |
| Old Faithful waiting times | Added 25 September 2026. The 272 waiting times of R’s faithful data. Left: density-scaled histogram with 3-minute bins from 40 to 97 minutes and a Gaussian kernel density estimate (SciPy gaussian_kde, Scott’s rule, bandwidth 4.43 minutes); the dashed line marks the mean, 70.9 minutes. Right: separate kernel estimates for the 97 waits after eruptions shorter than 3 minutes and the 175 waits after longer eruptions; each integrates to one. The chapter’s percentages count waits in [66, 76] minutes (16.9%) and [75, 85] minutes (43.0%), and use the 5% and 95% sample quantiles of each group (46–64 and 70–90 minutes). |
Iris data. R. A. Fisher’s Iris dataset at UCI, DOI 10.24432/C56C76, is distributed there under CC BY 4.0. These plots use the corrected Fisher values supplied by load_iris. The tree is fitted with scikit-learn’s decision-tree classifier.
Digit data. E. Alpaydin and C. Kaynak, Optical Recognition of Handwritten Digits (1998), UCI, distributed under CC BY 4.0. The figure uses the subset packaged by load_digits. Values are 8 × 8 block counts derived from digit images. The illustrative split described above is not the original UCI writer-separated benchmark; the data are not MNIST. The MNIST, CIFAR-10, and ImageNet error rates quoted in the chapter are published results, cited there and in Paper links; they were not recomputed.
Old Faithful data. The faithful dataset distributed with R’s datasets package: 272 eruptions of the Old Faithful geyser, Yellowstone National Park, with eruption length and waiting time to the next eruption in minutes. R’s documentation cites W. Härdle, Smoothing Techniques: With Implementation in S (Springer, 1991), and refers to the more complete version analyzed by A. Azzalini and A. W. Bowman, “A Look at Some Data on the Old Faithful Geyser”, Applied Statistics 39 (1990), 357–365. The values were exported unchanged from the rdatasets 0.2.10 Python package into Sources/Figure code/faithful.csv, keeping the columns eruptions and waiting.
Historical references. Campbell, Hoane, and Hsu, “Deep Blue,” Artificial Intelligence 134 (2002), 57–83, especially §§2 and 7.3, documents search and evaluation-weight tuning. LeCun, Bottou, Bengio, and Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE 86(11) (1998), 2278–2324, provides the historical convolutional-network example mentioned in the text; no figures from that paper are reproduced here.
Previously used figure
The earlier Neural network layers.svg remains in Sources. It is by Offnfopt, from Multi-Layer Neural Network-Vector.svg, released under CC0 and into the public domain.
Probability and statistics
On 26 September 2026 Probability and Statistics received twenty-nine drawn figures, one for most sections and subsections, and its seven earlier Matplotlib figures were redrawn in the same style under their original file names (marked Redrawn). Each figure is drawn from the stated model with D3 and typeset with KaTeX, then rendered to PNG at twice its CSS size by headless Chromium; the source of each is Sources/Figure code/js/src/<name>.js (see Reproducing the figures). Random draws use the seeded linear congruential generator in js/lib/prob.js, with Box–Muller Normal draws, so each figure is reproducible exactly. Their individual draws therefore differ from those of the NumPy examples in the chapter; the prediction-interval figure is the exception, and uses the data of its NumPy example. Simulation illustrations do not replace the assumptions or coverage guarantees stated in the notes.
| Local figure | Section | Construction |
|---|---|---|
| Relative frequencies | From relative frequencies to axioms | Relative frequencies of the events {1, 2}, {3} and {4, 5, 6} after n = 1, …, 5,000 rolls of a fair die (seed 5), stacked so that they fill [0, 1] at every n, on a logarithmic n axis. Dashed levels at 1/3 and 1/2 mark the limits of the band boundaries. |
| Random variable as a map | Random variables and their laws | Schematic of the number of heads X on Ω = {HH, HT, TH, TT}: arrows from each outcome to its value, the preimage {X = 1} = {HT, TH} outlined, and the law P_X drawn as bars of length 1/4, 1/2 and 1/4. |
| Density and CDF | Mass, density, and cumulative probability | Redrawn. Unif(0, 1/2) on matching horizontal scales: density 2, and the CDF 2z on (0, 1/2). The shaded area from 0 to 0.2 equals the marked CDF value 0.4. |
| Natural frequencies | Conditioning and dependence | Expected counts among 10,000 messages for 1% spam, flagging probabilities 0.90 (spam) and 0.05 (ordinary): 90 and 495 flagged, 90/585≈0.154. |
| Simpson's paradox | Simpson’s paradox | Success counts of Charig et al. (1986), doi:10.1136/bmj.292.6524.879: A 81/87 and 192/263, B 234/270 and 55/80 for small and large stones; overall 273/350 and 289/350. Dot areas grow with group size. Right: 87 small and 263 large stones among A’s patients, 270 and 80 among B’s. |
| Total variance | Conditional expectation and total variance | Y=2C+G, P(C=1)=0.3, G∼N(0,1): the weighted group densities 0.7φ(y) and 0.3φ(y−2) and their sum. Var Y=1+0.84. |
| Discrete families | Discrete observations | Left: Binomial(10, 0.4), Binomial(40, 0.1), and Poisson(4) mass functions, k=0,…,12. Right: Hypergeometric(N=20, K=8, n=10) against Binomial(10, 0.4). |
| Poisson process | Continuous observations and uncertain probabilities | One rate-1 run on [0, 8] from exponential gaps (seed 77): events at 1.32, 2.66, 3.27, 3.93, 4.28, 5.92, and 6.54; counts 1, 3, 2, 1 in the windows of length 2. |
| Gamma and beta shapes | Continuous observations and uncertain probabilities | Gamma(α, 1) densities for α=1, 2, 5; Beta densities for (½, ½), (1, 1), (2, 2), and (2, 6), drawn up to height 3.2. |
| Student t tails | Distributions derived from Normal samples | t densities for ν=1, 3, 10 and N(0,1); right: P(T>x) on a logarithmic axis for 0≤x≤6, with the t₁ tail in closed form and the others by Simpson’s rule. |
| Sums of uniforms | Sums and convolution | Exact densities of sums of 1, 2, and 3 independent Unif(0,1) variables (Irwin–Hall), with the Normal density of mean k/2 and variance k/12 dashed. |
| Change of variables | Change of variables | X∼N(0, 1.8²) and Y=1/(1+e^{−X}), with density p_X(logit y)/[y(1−y)]. Bands of probability 0.2 between the X quantiles 0.4 and 0.6 and between 0.78 and 0.98, traced through the curve. |
| Inverse-CDF sampling | The CDF transform and inverse-CDF sampling | Exp(1): levels u=0.05, 0.15, …, 0.95 mapped to −log(1−u); right: 5,000 transformed uniforms (seed 3) in 40 bins on [0, 5] with the density e^{−x}. |
| Order statistics | Order statistics | Beta(k, 6−k) densities of the order statistics of five Unif(0,1) observations, with the means k/6 marked. |
| Gaussian conditioning | Gaussian vectors and conditioning | Standard bivariate Normal with correlation 0.7; contour levels 0.005, 0.02, 0.045, 0.08, 0.12, 0.17. Slice at x=1.2: conditional N(0.84, 0.51) against the marginal N(0, 1). |
| Running averages | Laws of large numbers and the central limit theorem | Five running averages of Exp(1) draws and five of standard Cauchy draws (seed 12), n=1,…,5,000 on a logarithmic axis. Left band: 1±2/√n. |
| Standardized means | Laws of large numbers and the central limit theorem | Redrawn. For n=1, 5, 30: √n(Z̄−1) from 50,000 datasets of n iid Exp(1) observations (seed 20260921), 74 bins on [−3.2, 4.2] with heights normalized by the full count; the exact density from Gamma(n, 1), dashed; the N(0,1) density. |
| Grid and Monte Carlo | Monte Carlo integration | ∫∏ⱼe^{−uⱼ²}du over [0,1]^d for d=1 and 10: relative error of the midpoint grid with m points per coordinate (N=m^d, up to 2×10⁵ for d=1 and 59,049 for d=10), and the Monte Carlo relative root-mean-square error √((J/I²)^d−1)/√N, where I=∫₀¹e^{−u²}du and J=∫₀¹e^{−2u²}du. |
| Empirical CDF | Empirical distributions and the plug-in principle | Redrawn. Left: right-continuous empirical CDF of (2, 4, 4, 7, 10), filled points at the values and open points at the left limits. Right: empirical CDF of 100 N(0,1) draws (seed 5), the 95% DKW band of half-width √(log 40/200)≈0.136 clipped to [0, 1], and Φ. |
| Bias and variance | Bias, variance, and mean squared error | Exact MSE of K/n and (K+1)/(n+2) for K∼Binomial(20, p). They cross where p(1−p)=20/164, at p≈0.142 and 0.858. |
| Delta method | Consistency, asymptotic normality, and the delta method | p̂∼N(0.2, 0.2·0.8/50) and g(p)=log[p/(1−p)]: exact density of g(p̂) by change of variables against the delta-method N(−1.386, 6.25²·0.0032). The shaded strip is 0.2±2 SD(p̂). |
| Likelihood | Likelihood and maximum likelihood | Redrawn. Bernoulli likelihoods for 8/10 and 80/100 successes divided by their maxima; right: their logarithms with the quadratic approximations −n(p−0.8)²/(2·0.16), dashed. |
| Interval coverage | Confidence intervals and repeated-sampling coverage | Redrawn. 40 samples of size 20 from N(0,1) (seed 3), each with its 95% Student t interval (t₁₉ quantile 2.0930); 38 cover 0. |
| Proportion coverage | Proportions and score inversion | Exact coverage of the nominal 95% Wald and Wilson intervals for n=30, summing binomial probabilities at 1,999 values of p. Wilson’s minimum, about 0.84, occurs near p=0.006; for 0.05≤p≤0.95 Wilson stays above 0.92 and Wald falls to 0.78. |
| Bootstrap | Bootstrap approximation of sampling uncertainty | Means of 30 Exp(1) observations over 20,000 fresh datasets, and 20,000 bootstrap means from one dataset (seed 13, mean 1.10). Both standard deviations are 0.182; the population value is 1/√30≈0.183. |
| p-value | What a p-value states | N(0,1) null density; level-0.05 two-sided rejection region beyond ±1.96 and the p-value 2[1−Φ(2.10)]≈0.036. |
| Power | Effect sizes and sample size | N(0,1) and N(2.5,1) with the two-sided level-0.05 rejection region; power 1−Φ(1.96−2.5)+Φ(−1.96−2.5)≈0.705. |
| Multiple testing | Multiple comparisons and selection | 30 uniform p-values and 10 from P(G>Z) with Z∼N(3,1) (seed 4), sorted, on a logarithmic axis; Benjamini–Hochberg at q=0.1 rejects 8, all real effects, and Bonferroni at 0.1/40 rejects 5. |
| Bayesian update | Bernoulli observations and a Beta prior | Redrawn. Beta(2,2) prior and Beta(10,4) posterior on one density scale, with the equal-tailed 95% interval [0.4619, 0.9091] shaded; the relative likelihood for 8 successes in 10 trials on its own panel. |
| Normal shrinkage | Prediction and shrinkage | Prior N(0,1), σ=2, n=4, sample mean 1.5: likelihood N(1.5, 1) in μ and posterior N(0.75, 0.5); right: the posterior predictive N(0.75, 4.5). |
| Cost threshold | Decisions, loss, and risk | Expected costs 1−p of predicting 1 and 4p of predicting 0 (c_FP=1, c_FN=4); they cross at p=1/5. |
| Prediction interval | Estimating a mean versus predicting an observation | Redrawn. The data of the chapter’s NumPy example (seed 7): n=60, Y=1+2u+N(0,1) at equally spaced u in [−2, 2], OLS with intercept, and pointwise 95% intervals for the mean response and a new observation (t₅₈ quantile 2.0017). |
| Tail bounds | Appendix C | Left: the tails of W ∼ Binomial(1000, 0.01) on a logarithmic axis, P(W ≤ k) below the mean 10 and P(W ≥ k) above it, with the exact binomial sums at integer k, the Normal approximation with variance 9.9, Hoeffding’s bound exp(−2(k − 10)²/1000), and the Chernoff bound exp(−1000 D(k/1000 ‖ 0.01)) from the exact binomial MGF, where D is the Bernoulli Kullback–Leibler divergence. Right: the Chernoff exponent t − 10 log(1 + t/10) of a centered sum of ten Exp(1) variables on logarithmic axes, with the Gaussian exponent t²/20 and the line t; the regime boundary is drawn at t = 10. |
| Wald, likelihood ratio, and score | Appendix F | Bernoulli log-likelihood for 14 successes in 20 trials minus its maximum, θ̂=0.7, θ₀=0.5: gap 0.2, drop 1.65, and score 14/0.5−6/0.5=16. |
| Leverage and influence | Appendix G | Twenty-four ordinary points with x evenly spread on [0, 6] and y = 1 + 0.8x + N(0, 0.6²) (seed 21), and three unusual points: A at x = 12, 0.1 above the least-squares line of the ordinary points; B at x = 3.2, 3.4 above the line 1 + 0.8x; C at (11, 4.2). Left: the least-squares line of the ordinary points and the line refitted with C added. Right: leverage and internally studentized residual of each ordinary point in the fit of the ordinary points, and of each unusual point in the fit of the ordinary points plus that point alone, with Cook’s distance contours D = 0.5 and 1 for d = 2. |
| James–Stein risk | Appendix H | Risk of X∼N(θ, I₁₀) and of (1−8/‖X‖²)X under squared-error loss: 10, and 10−64·E[1/(8+2K)] with K∼Poisson(‖θ‖²/2). |
Survey of existing figures
Existing figures were checked before any were drawn: Wikimedia Commons for Bayes’ theorem, Simpson’s paradox, the Poisson process, the empirical CDF, the law of large numbers, the central limit theorem, p-values, and the Wald, score, and likelihood-ratio comparison, and the conditioning figure of Distill’s A Visual Exploration of Gaussian Processes. The Commons candidates were statistical-software plots or densely annotated diagrams, and the Distill figure opens with an uncorrelated distribution and shows the same standard deviation for the conditional and the marginal law. Those topics were drawn instead.
Calculus and optimization
The original figure below accompanies Calculus and Optimization. It is computed directly from the stated function with NumPy and Matplotlib and has an opaque white background.
| Local figure | Function and interpretation |
|---|---|
| forward tangent | For f(x)=x1x2+sin(x1x2), x=(2,3), and v=(1,-1), the actual output along x+tauv is compared with its first-order approximation f(x)+tauDf(x)[v]. The forward tangent is the slope 1+cos(6), approximately 1.96017, at tau=0. |
Drawn figures, 26 September 2026
These original figures were added on 26 September 2026, one for most sections and subsections of the chapter; the last two rows replace the Matplotlib versions of the optimizer-path figures drawn on 23 September 2026. Each is drawn from the stated function and update rule with D3 and typeset with KaTeX, then rendered to PNG at twice its CSS size by headless Chromium; the source of each is Sources/Figure code/js/src/<name>.js (see Reproducing the figures). They share one palette: orange, blue, and magenta pass the lightness, chroma, color-vision-deficiency, and 3:1 contrast checks as a set on white, and labels use gray and near-black. Random draws use a seeded linear congruential generator, so each figure is reproducible exactly.
| Local figure | Section | Construction |
|---|---|---|
| Tanh layer graph | Example: one layer with a tanh activation | Schematic of z=Wx+b, a=tanh z, ℓ=‖a−y‖²/2 with the gradients derived in the section. |
| Local derivative rules | Computational graphs and local derivative rules | Schematic of one node z=φ(a,b) under the forward-mode and reverse-mode rules of the chapter, with the product z=ab as example. |
| Autograd graph | Backpropagation in PyTorch | Node names read from loss.grad_fn and next_functions for the chapter’s tanh-layer code, run with PyTorch 2.14.0 (CPU). |
| Jacobian by each mode | Choosing between forward and reverse mode | Schematic Jacobians with m=5 and d=7, and a single row of length 9 for a scalar output. |
| Quadratic forms | Curvature and Taylor’s theorem | Contours of hᵀHh/2 on [−1.6, 1.6]² for H=R diag(λ1, λ2) Rᵀ, R a rotation by 30°, with (λ1, λ2)=(2.4, 0.7), (1.6, −1), and (2, 0). Levels λ1(0.24k)²/2, k=1,…,9, for the two semidefinite cases; ±(0.24k)²/2 and 0 for the indefinite case. |
| Taylor models | The quadratic approximation | f(x)=sin x+x/2 at x=0.6 with its first- and second-order Taylor models; right panel: |f(x+h)−model| for 10⁻³≤h≤1 on logarithmic axes. |
| Minimizers and infimum | Optimization problems and notation | Left: a monotone cubic interpolant (D3 curveMonotoneX) of eleven hand-placed points, with a flat stretch at height 0.45 on [1.35, 2.35]. Right: eˣ on [−4, 1.3]. |
| Convexity and strong convexity | Convex and strongly convex functions | Left: f(x)=log cosh(1.1x)+0.12x, chord between −2.2 and 1.6, tangent at 0.9. Right: f(y)=0.6 log cosh(1.5(y−0.5))+0.25(y−0.5)², with f’‘≥0.5=μ, minimizer 0.5, and f⋆=0. At x=2.5, f’(x)≈1.896; the lower bound q_x is minimized at y≈−1.291 with value ≈−1.208. |
| Lipschitz envelopes | Three different Lipschitz assumptions | f(y)=sin(1.2y)+0.3y at x=0.3, with G=max|f’|=1.5, L=max|f’‘|=1.44, and ρ=max|f’’‘|=1.728. Shaded regions: |f(y)−f(x)|≤G|y−x|; |f(y)−f(x)−f’(x)(y−x)|≤L(y−x)²/2; |f(y)−T₂(y)|≤ρ|y−x|³/6, where T₂ is the second-order Taylor polynomial. |
| Gradient step as model minimizer | Gradient descent and step sizes | f(x)=log cosh x+0.1x², with f’’≤1.2=L, from x_t=0.9, where f(x_t)≈0.4408. With η=0.5/L, 1/L, and 2.5/L the step reaches 0.527, 0.153, and −0.967, where f≈0.1604, 0.0140, and 0.5027. |
| Descent-lemma guarantee | The descent lemma | f(x)=log cosh x, L=1, x=1.5, ∇f(x)=tanh 1.5≈0.905. Solid: f(x)−f(x−η∇f(x)); dashed: η(1−Lη/2)‖∇f(x)‖², for 0≤η≤2.6. |
| Armijo backtracking | Backtracking when the smoothness constant is unknown | A quadratic stand-in φ(η)=f(x+ηp)=1−1.6η+2.4η², so f(x)=1 and ∇f(x)ᵀp=−1.6; c=0.3, τ=1/2, η₀=1. The condition holds exactly for η≤0.467, so 1 and 0.5 are rejected and 0.25 is accepted. |
| Nonconvex gradient descent | Smooth functions without convexity | f(x,y)=(x²−1)²+0.3x+1.2y², stationary points at y=0 and x≈−1.036 (global minimizer), 0.075 (saddle), and 0.961 (local minimizer). η=0.02 for 900 updates from (−1.5, 1), (−0.6, 1.08), (0.078, 1.08), (1.5, −0.98), and (0.55, −1.02). |
| Distance to the minimizer | Smooth convex functions | f(x)=log cosh(1.2x₁)+log cosh(3x₂)/2, L=4.5, η=1/L, 40 updates from (4.6, 1.9). Circles through the iterates t=0, 3, 6, 10, and 15. |
| GD and SGD paths | Stochastic gradients and convergence in expectation | f(x)=(x₁²+6x₂²)/2, η=0.12, 120 updates from (3.3, 1.5). SGD adds independent Normal(0, 2.2²) noise to each gradient coordinate (Box–Muller, seed 7). |
| Heavy ball and Nesterov | Nesterov acceleration | f(x)=(0.35x₁²+3x₂²)/2, x_{t−1}=(−3.2, 1.1), x_t=(−2, 0.2), coefficient 0.8, step 0.5. Heavy ball: x_t+0.8(x_t−x_{t−1})−0.5∇f(x_t). Accelerated: y_t=x_t+0.8(x_t−x_{t−1}), then y_t−0.5∇f(y_t). |
| AdaGrad and RMSProp steps | AdaGrad and RMSProp | One coordinate with |g_t|=s_t u_t, s_t=1 for t<200 and 0.1 afterward, u_t uniform on [0.6, 1.4], random signs (seed 11). η=0.01, ε=10⁻⁸, v₀=0; RMSProp uses β₂=0.97 without bias correction, as displayed in the chapter. Step lengths η|g_t|/(√v_{t+1}+ε) on a logarithmic axis. |
| Adam bias correction | Adam and bias correction | Constant gradient g=1, β₁=0.9, β₂=0.999: m=1−β₁ᵏ and v=1−β₂ᵏ after k updates. The uncorrected ratio m/√v peaks at about 6.57 at k=12 and is 1.26 at k=1000. |
| Adam with L2 versus AdamW | L2 regularization and AdamW | Two parameters from 1 with loss gradients Normal(0, 1) and Normal(0, 10⁻⁶) (seed 5); η=10⁻³, λ=1, β₁=0.9, β₂=0.999, ε=10⁻⁸, bias-corrected moments, 3,000 updates, plotted every 10. Adam with L2 adds λx to the gradient; AdamW applies (1−ηλ)x separately. Dashed: e^{−ηλt}. |
| Learning-rate schedules | Schedules, batches, and clipping | T=1,000 updates. Step decay multiplies by 0.1 at updates 400 and 800. Both warmups are linear over 60 updates; cosine decays to zero at T; inverse square root is √(60/t). |
| Rate shapes | Convergence rates and their scope | The sequences 1/√T, 1/T, 1/T², 0.8ᵀ (κ=5), and e_{t+1}=e_t² from e₁=0.9, for T=1,…,60, on a logarithmic axis. |
| Optimizer paths | Optimizer paths on a quadratic | Redrawn. The recurrences of Appendix H: f(x)=(x₁²+20x₂²)/2 from (3, 1), 40 exact-gradient updates, buffers starting at zero. Gradient descent η=0.09; unnormalized momentum η=0.09, β=0.65; smooth FISTA η=0.05, a₀=1; Adam η=0.18, β₁=0.9, β₂=0.99, ε=10⁻⁸, no decay. The same contour levels (0.002, 0.01, 0.03, 0.08, 0.2, 0.5, 1, 2, 4, 8, 14, 22) in every panel. Each strip magnifies x₁∈[−0.65, 0.95], x₂∈[−0.26, 0.26] by about 2.9. Final objective values: 2.38×10⁻³, 2.33×10⁻⁷, 1.33×10⁻³, and 0.119. |
| Optimizer objective traces | Optimizer paths on a quadratic | Redrawn. The same runs, evaluated as xᵀdiag(1, 20)x/2 at t=0,…,40 on a logarithmic axis; the minimum value is zero. Finite illustrative runs, not a benchmark or a general convergence claim. |
Imported figures, 26 September 2026
These figures were retrieved on 26 September 2026 and are used under the licenses shown. The Wikimedia files are unmodified. The four Distill figures are static captures of interactive figures, with the changes listed; Sources/Figure code/ch03_capture_distill.py repeats the capture. The survey figure is cropped from the published PDF by ch03_crop_baydin.py in the same folder.
| Local figure | Source and creator | License and changes | Use in the chapter |
|---|---|---|---|
| Differentiation approaches | Figure 2 of A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind, Automatic Differentiation in Machine Learning: a Survey, JMLR 18(153):1–43, 2018 | CC BY 4.0. Page 5 rendered at 300 dpi and cropped above the caption; artwork unchanged. | Automatic differentiation |
| Forward accumulation | ForwardAccumulationAutomaticDifferentiation.png, Berland at English Wikipedia, 18 January 2007; drawn with TikZ | Public domain, released by the author. Unmodified. | Forward mode |
| Reverse accumulation | ReverseaccumulationAD.png, Berland at English Wikipedia, 18 January 2007; drawn with TikZ | Public domain, released by the author. Unmodified. | Reverse mode |
| Saddle point | Saddle point.svg, Nicoguaro, 17 May 2016 | CC BY 3.0. Unmodified. The surface is z=x²−y². | Optimality conditions |
| Newton and gradient descent | Newton optimization vs grad descent.svg, Oleg Alexandrov, 23 June 2007; Matlab and Inkscape | Public domain, released by the author. Unmodified. | Newton’s method |
| Subderivative | Subderivative illustration.png, Maksim at English Wikipedia, uploaded 20 March 2006; the file page includes its Matlab source | Public domain, released by the author. Unmodified. The piecewise-linear function passes through (0, 0.2), (1, 0.5), and (2, 1.5); the red lines through x₀=1 have slopes 0.5 and 0.8. | Constraints and nonsmooth objectives |
| Gradient descent in the eigenbasis | Gabriel Goh, Why Momentum Really Works, Distill, 2017, doi:10.23915/distill.00006; figure in “First Steps: Gradient Descent” | CC BY 2.0. Hand glyph on the draggable start point hidden; default parameters. | Quadratics expose the stability threshold |
| Eigencomponent convergence | Same article, gradient-descent milestones figure | CC BY 2.0. Step-size slider set to 1 (=1/λ_max); explanatory text, slider, and in-figure notes hidden; margins cropped. | Smooth strongly convex functions |
| SGD phases | Same article, stochastic-gradient figure | CC BY 2.0. Momentum set to 0 and step size to about 1.9 with the two-dimensional slider; explanatory text, slider, and in-figure note hidden; margins cropped. The single run is redrawn on each visit, so its small dots differ between captures. | A basic stochastic convergence bound |
| Momentum on a valley | Same article, opening figure | CC BY 2.0. Default step size 0.02 and momentum 0.99; sliders and caption omitted; hand glyph hidden. | Momentum as a gradient filter |
Goh’s article writes w for the parameters and α for the step size; its momentum recurrence z^{k+1}=βz^k+∇f(w^k), w^{k+1}=w^k−αz^{k+1} is the chapter’s unnormalized gradient-buffer convention.
Survey of existing figures
Existing figures were checked before any were drawn: Wikimedia Commons, Distill’s momentum article, the CS231n notes (MIT license), Dive into Deep Learning (CC BY-SA 4.0), and the Baydin et al. survey. Existing figures were used where one matched the chapter’s content and notation. The CS231n Nesterov diagram depicts the momentum-buffer form of Nesterov’s method rather than the chapter’s extrapolation recurrence, and the Commons and Dive into Deep Learning candidates for convexity, Lipschitz continuity, Taylor approximation, gradients, and stochastic paths were plain or small; those topics were drawn instead.
Information and learning theory
On 27 September 2026 Information and Learning Theory received nine drawn figures, and its four earlier Matplotlib figures were redrawn in the same style under their original file names (marked Redrawn). They are drawn with D3 and KaTeX as described under Probability and statistics, from the sources Sources/Figure code/js/src/<name>.js. Information quantities are in bits, following the chapter’s convention; one bit equals ln 2 nats.
| Local figure | Section | Construction |
|---|---|---|
| Surprisal and binary entropy | Surprise and entropy | Redrawn. Left: surprisal −log₂ p in bits, with outcomes of probability 1/2, 1/4, and 1/8 marked at 1, 2, and 3 bits. Right: binary entropy h_B(r) in bits, with maximum one bit at r = 1/2 and zero at the deterministic endpoints. |
| Prefix code tree | Codes, compression, and typical sequences | The prefix code a→0, b→10, c→110, d→111 as paths in a binary tree, with 0 for the upper branch. The bar splits [0, 1) into the fractions 2^−ℓ = 1/2, 1/4, 1/8, 1/8 claimed by the codewords. |
| Typical set | Codes, compression, and typical sequences | n = 100 independent Bernoulli(0.1) bits. Left, on a base-2 logarithmic axis: the probability of one sequence with k ones and the number of such sequences, whose product is P(K = k). Right: P(K = k) for k ≤ 30, with the typical set for ε = 0.13 bits shaded (entropy 0.469 bits; 6 ≤ k ≤ 14, probability 0.870). The all-zero sequence has probability 0.9¹⁰⁰ ≈ 2.66×10⁻⁵. |
| Forward and reverse KL | Cross-entropy, divergence, and log loss | p = ½N(−2, 0.6²) + ½N(2, 0.6²). Over Gaussian q, D_KL(p‖q) is minimized by matching the mean 0 and variance 4.36 of p. D_KL(q‖p), minimized by grid search over the mean and standard deviation, is smallest at N(2.0, 0.605²), near one component; its mirror image does equally well. |
| Cross-entropy and KL | Cross-entropy, divergence, and log loss | Redrawn. True Bernoulli success probability r = 0.2, candidate s. In bits: the entropy is 0.7219; at s = 0.6 the cross-entropy is 1.2050 and KL is 0.4830. The displayed vertical range ends at three bits; both endpoint limits diverge. |
| Information diagram | Mutual information and data processing | A fair bit X and Y = X xor N, with N ∼ Bernoulli(0.1) independent of X, in bits: H(X) = H(Y) = 1, H(X∣Y) = H(Y∣X) = h(0.1) ≈ 0.469, I(X;Y) ≈ 0.531, and H(X,Y) ≈ 1.469. |
| Maximum entropy | Continuous variables and maximum entropy | Three densities with mean 0 and variance 1: Gaussian, Laplace with scale 1/√2, and uniform on [−√3, √3]. Their differential entropies are 2.047, 1.943, and 1.792 bits. |
| Markov entropy rate | Sequences, entropy rates, and language-model loss | Redrawn. Left: 48 steps of a symmetric binary chain started from its uniform stationary law, for switching probabilities η = 0.1 and η = 0.5 (seed 3). Right: entropy rate h_B(η) in bits for 0 ≤ η ≤ 1/2, against the one-bit marginal entropy; the rate is 0.469 bits at η = 0.1. At η = 0 the stationary process is a uniform mixture of two constant sequences, so ergodicity is not assumed at that endpoint. |
| Excess-risk decomposition | The best possible rule and the best available rule | Schematic, not to scale: all rules, the class H, the Bayes rule outside H, the best member h_H of H, the exact empirical risk minimizer, and the returned rule within the optimization tolerance. |
| Empirical and population risk | From concentration to generalization | Redrawn. X is uniform on the integers 0, …, 100; the success probability is 0.15 below 55 and 0.85 from 55 onward. Threshold rules 1{x ≥ t}. The sample of size 80 drawn by NumPy’s default_rng(7) in the chapter’s code, embedded in the source, selects t = 53 with empirical risk 0.125 and exact population risk 0.16386. The population optimum is t = 55 with risk 0.15. This is one sample, not a universal curve shape. |
| Growth functions | Infinite classes and VC dimension | Label patterns on m ≤ 16 points, on a base-2 logarithmic axis: all 2^m labelings; thresholds, m + 1; intervals, 1 + m(m + 1)/2; and affine halfplanes on points in general position, m² − m + 2 (Cover’s count). |
| Margin losses | Norm bounds for linear scores | The zero–one loss 1{s ≤ 0}, the ramp loss min{1, max{0, 1 − s/ρ}}, and the hinge loss max{0, 1 − s/ρ}, against the margin s = y f(x) in units of ρ. |
| Capacity and rate–distortion | Appendix A, Lossy compression and rate–distortion | Left: capacity 1 − h(η) of the binary symmetric channel with flip probability η, in bits per use. Right: R(D) = 1 − h(D) for 0 ≤ D ≤ 1/2 and zero beyond, for a fair binary source with Hamming distortion, in bits per symbol. |
Sourced learning-theory diagrams
The four VC illustrations are unmodified Wikimedia Commons SVGs, retrieved 23 September 2026. Their backgrounds remain white in Obsidian.
| Local figure | Source and creator | License and interpretation |
|---|---|---|
| VC labeling 1 | VC1.svg, MithrandirMage, based on BAxelrod; 23 June 2012. Current simplified SVG by Jarvisa, 26 June 2019. | CC BY-SA 3.0. One linearly separable labeling of the three points. |
| VC labeling 2 | VC2.svg, MithrandirMage, based on BAxelrod; 23 June 2012. Current simplified SVG by Jarvisa, 26 June 2019. | CC BY-SA 3.0. A second labeling of the same three points. |
| VC labeling 3 | VC3.svg, MithrandirMage, based on BAxelrod; 23 June 2012. | CC BY-SA 3.0. A third labeling; the three examples together do not exhaust all eight labelings. |
| Four-point obstruction | VC4.svg, MithrandirMage, based on BAxelrod; 23 June 2012. Current simplified SVG by Jarvisa, 26 June 2019. | CC BY-SA 3.0. The alternating labeling of a convex quadrilateral cannot be separated by one line. This is one obstruction, not by itself a proof for every four-point set; the chapter gives the general Radon argument. |
These illustrations use Creative Commons Attribution-ShareAlike 3.0 Unported, whose license is linked below with the SVD illustration.
Numerical computing
On 27 September 2026 Numerical Computing with NumPy and PyTorch received six drawn figures, made in the same way. The training-curve figure plots values computed by Sources/Figure code/ch06_training_curve.py, which reruns the chapter’s training code; its final values agree with the saved output of the companion notebook in the environment of the computing setup.
| Local figure | Section | Construction |
|---|---|---|
| Strides and views | Indexing, shared storage, and rearrangement | X = arange(6).reshape(2, 3) in float64 and row-major order, with byte offsets 0 to 40. X has strides (24, 8), X.T has strides (8, 24), and X.reshape(3, 2) has strides (16, 8). Cells are shaded by value, and arrows follow memory order. |
| Broadcasting | Broadcasting and reductions | The shape pairs (4, 3) with (3,), (4, 3) with (4, 1), and (4, 1) with (4,), which broadcasts to (4, 4). Dashed cells are entries read again along a length-one or missing axis. |
| Masked averages | Masks and the denominator of an average | Three sequences of lengths 5, 2, and 3 stored in six positions. Each real token has weight 1/10 in the token mean, and 1/15, 1/6, or 1/9 in the mean of per-sequence means. |
| Range of the exponential | Stable probability calculations | The interval −745.13 < z ≤ 709.78 on which e^z is a positive finite float64, with the chapter’s logit vectors (1000, 1001, 999) and (−1000, 0, 1000) before and after subtracting their maxima. |
| Training curves | A complete numerical example | The chapter’s standardized logistic regression: 769 training and 257 test observations, batch size 64, SGD with learning rate 0.1, 40 epochs, seeds 7, 11, and 17. Mean log loss and accuracy of both sets after each epoch, and the loss of every mini-batch at its update; single-example batch losses above 1.3 are marked by triangles at the top edge. Computed with PyTorch 2.14.0 on a CPU. The dashed lines are the population log loss 0.5197 nats and accuracy 0.7397 of the true model, by numerical integration over its logit s ∼ N(−0.3, 2.33). |
| Floating-point spacing | Appendix A, Representable numbers | Top: every value of a toy binary format with p = 3 and exponents −2 to 2: the subnormal values 1/16, 1/8, and 3/16, the normal values m·2^(e−2) with m = 4, …, 7, and the largest value 7. Bottom: relative spacing against magnitude, log–log, for float64, float32, bfloat16, and float16, computed from their precision, smallest normal, and largest finite values. Bands between ε/2 and ε cover the normal ranges; dashed lines cover the subnormal ranges, where relative spacing rises to 1 at the smallest subnormal value. |
Linear algebra
The chapter uses the Wikimedia SVD and eigenvector-shear illustrations, the original point–subspace diagram and PCA illustration, and a figure drawn on 26 September 2026. The Wikimedia artwork remains unchanged; the SVG files receive a white background from the existing Obsidian display rule. The original figures have opaque white backgrounds.
| Local figure | Original work and creator | License used | Reading the figure |
|---|---|---|---|
| Point and subspace projection | Original illustration for these notes, 23 September 2026. | No external artwork | The plane is the subspace z=0 in R³. The point b=(1.1,0,1.8) projects to p=(1.1,0,0), so b−p=(0,0,1.8) is orthogonal to both plane directions. A fixed oblique view maps (x,y,z) to (x+0.65y,0.45y+z). The origin lies in the plane; the shaded parallelogram represents a portion of the infinite subspace. |
| SVD geometry | Singular-Value-Decomposition.svg, Georg-Johann, 28 August 2010; current revision 31 August 2010. | CC BY-SA 3.0 | A factorization of the real shear matrix M=[[1,1],[0,1]]. The figure uses M in place of the chapter’s A and V* in place of Vᵀ; these adjoint conventions coincide over the reals. Colored arrows track the canonical basis vectors, not the singular vectors. The blue semiaxes are marked with singular values. |
| PCA and rank-one reconstruction | Original figure for these notes. | No external artwork | Synthetic sample of 120 observations (NumPy default_rng seed 23): independent normal coordinates with standard deviations 0.75 and 0.24, rotated by 48 degrees and translated by (2.5,2.5). The displayed principal directions are fitted by centered SVD. Left: observations and rank-one reconstructions in original coordinates. Right: principal-component scores with the second score set to zero for reconstruction. Retained variance is approximately 0.906999. The four-point NumPy example in the chapter is a separate dataset. |
| Eigenvector of a shear | Mona Lisa eigenvector grid.png, TreyGreer62, 21 January 2011, made from the public-domain painting. | CC0 1.0 | A horizontal shear of the painting and its grid. The blue arrow lies along an eigenvector with eigenvalue 1; the red arrow changes direction. Retrieved 26 September 2026. |
| Conditioning of a 2×2 system | Original figure drawn for these notes, 26 September 2026; source Sources/Figure code/js/src/linear-algebra-conditioning.js. | No external artwork | Left: A=[[1, 1], [1, −1]], κ₂=1. Right: A=[[1, 1], [1, 1.15]], κ₂≈28.8. In both, b=A(1, 1)ᵀ and b₂ is raised by 0.15, moving the solution to (1.075, 0.925) and to (0, 2). The relative change ‖δx‖₂/‖x‖₂ divided by ‖δb‖₂/‖b‖₂ is 1.00 and 19.6. |
The SVD image is used under Creative Commons Attribution-ShareAlike 3.0 Unported; the eigenvector-shear image is dedicated to the public domain under CC0 1.0.
Credits for earlier Wikimedia figures retained in Sources/Images
These files are no longer embedded in the chapter. Their artwork is unchanged.
| Local figure | Original work and creator | License used | Reading the figure |
|---|---|---|---|
| Unit balls | 2D unit balls.svg, Waldir Pimenta (Waldyrious), based on MinkowskiCircles.svg; original work dated 21 March 2014, current revision 7 May 2021. | CC BY-SA 3.0 | The p=1, 2 and infinity panels show norm unit balls. The p=0.25 and p=0.5 panels are not norm balls: the triangle inequality fails below p=1. |
| Orthogonal projection | Orthogonal projection.svg, Jitse Niesen, 1 November 2006. | Public domain, released by the author | P maps each point onto the line m along a perpendicular. A point w on the line is fixed. In the chapter, m corresponds to the subspace S, Pb=p, and b−Pb=r. |
| Principal directions | GaussianScatterPCA.svg, Nicoguaro, 8 February 2016; current file revision 10 February 2016. | CC BY 4.0 | A separate illustrative bivariate Gaussian sample, not the four-row dataset in the chapter’s NumPy example. The arrows show the distribution’s principal directions; the mean is (1,3), and the population standard deviations along those directions are 3 and 1. |
The PCA illustration is licensed under Creative Commons Attribution 4.0 International; the unit-ball illustration uses CC BY-SA 3.0, linked above. The projection illustration was released into the public domain by its author. File pages were retrieved 23 September 2026.
Reproducing the figures
The tables above record the models and sources used for the illustrations. The calculus appendix Code for optimizer trajectories and simplified plots reproduces its numerical paths and simplified plots; it does not reproduce every styling detail of the embedded figures. Preserve the stated model, seed, units, and interpretation when regenerating a figure.
The figures added from 25 to 27 September 2026 have complete sources in Sources/Figure code:
| Script | Figures and printed values |
|---|---|
ch01_terminology.py | The two iris-tree figures, the clear and ambiguous digits, and the Old Faithful densities. It prints the tree’s thresholds, errors, and example predictions; the digit probabilities; the held-out accuracies of the network, nearest neighbor, logistic regression, and k-means; the k-means cluster compositions; and the Old Faithful summaries quoted in chapter 1. |
foundfig.py | Shared palette, fonts, and export for ch01_terminology.py: opaque white background, 150 dpi PNG cropped to its contents, written to Sources/Images. |
faithful.csv | The Old Faithful data described above. |
js/src/*.js | The drawn figures of chapters 2 to 6, one file per figure, named after its PNG. js/lib/fig.js and js/lib/base.css hold the shared helpers, palette, and fonts; js/lib/optruns.js holds the optimizer runs of the calculus Appendix H; and js/lib/prob.js holds the seeded random-number generator and the distribution functions of chapter 4. build.sh wraps each file into js/build/<name>.html, and render.js renders it at twice its CSS size and reports any overlapping or clipped labels and any label crossed by a drawn line. |
ch06_training_curve.py | The data of the chapter 6 training-curve figure. It reruns the chapter’s training code, records the loss and accuracy of both sets after each epoch and every mini-batch loss, and prints the values embedded in js/src/numerical-training-curve.js together with the final values quoted in the chapter. Needs PyTorch and SciPy. |
ch03_capture_distill.py | The four Distill captures, with the changes listed above. Needs network access to distill.pub. |
ch03_crop_baydin.py | The figure from Baydin et al., rendered from the published PDF. |
The Python scripts were run with Python 3.12.3, NumPy 1.26.4, SciPy 1.11.4, Matplotlib 3.8.4, and scikit-learn 1.4.2. The drawn figures were rendered with Node.js 22, D3 7.9.0, KaTeX 0.16.47, the Roboto and Roboto Mono fonts from Fontsource 5.3.0, and Playwright 1.56.0 with its Chromium build; the captures and the crop also used Python Playwright 1.56.0, Pillow 12.2.0, and Poppler’s pdftoppm 24.02. The computing setup gives the commands. Running a script overwrites the corresponding image files in Sources/Images. ch06_training_curve.py was run with Python 3.11.15, PyTorch 2.14.0 (CPU), and SciPy 1.17.1. Full plotting scripts are not bundled for the other authored images.