Reading plan

Work in progress: this part of the notes is still being revised.

Courses

The order below follows the chapters of this module. Each block lists the material to cover and its primary lectures; further reading and PyTorch examples provide additional depth. Blocks 1–12 form the main plan; 13–15 are optional extensions. Timing is flexible.

Foundations covered reverse-mode differentiation, backpropagation as an algorithm, and the PyTorch tensor, autograd, and module interfaces; ML covered linear models, losses, regularization, and model selection. Review those chapters as needed in place of the corresponding introductory lectures (UMich lectures 2, 3 and 6; UNIGE modules 1–2). Deep generative models belong to the Generative AI module, language models to NLP and LLMs, and deep reinforcement learning to RL.

Topic list

Reference books used below

1. Deep feedforward networks {#1-deep-feedforward-networks}

Topics: From linear models to multilayer perceptrons; activation functions; the network as a composition of affine maps and nonlinearities; output layers and losses as likelihoods; universal approximation; the benefits of depth, including linear regions and depth-separation results; parameter counting.

Further reading: UDL chapters 3–5 develop shallow networks, deep networks, and losses as likelihoods, with interactive figures. DLB chapter 6 covers the same ground with more history. Foundations’ calculus chapter contains the backpropagation derivation used here.

2. Initialization and signal propagation {#2-initialization-and-signal-propagation}

Topics: How activations and gradients change in scale with depth; vanishing and exploding gradients; variance-preserving initialization (Glorot and He); saturation and dead units; activation statistics as a diagnostic; orthogonal initialization.

Further reading: UDL chapter 7 derives the He initialization and discusses exploding gradients.

3. Optimization for deep networks {#3-optimization-for-deep-networks}

Topics: Minibatch gradient noise; momentum and Nesterov momentum; adaptive methods and AdamW; learning-rate schedules, warmup, and decay; batch size and the learning rate; gradient clipping; the geometry of neural loss landscapes.

Further reading: UDL chapter 6 covers the optimizers; DLB chapter 8 discusses the obstacles specific to neural networks. Foundations’ calculus chapter covers convergence of gradient descent, momentum, and Adam on convex problems.

4. Normalization and residual connections {#4-normalization-and-residual-connections}

Topics: Batch normalization in training and evaluation; layer, group, and RMS normalization; residual connections and why they make depth trainable; pre-normalization and post-normalization; the residual network family.

Further reading: UDL chapter 11 treats residual networks and normalization together.

5. Regularization and generalization {#5-regularization-and-generalization}

Topics: Weight decay; dropout; data augmentation and mixup; early stopping; label smoothing; ensembles and weight averaging; memorization of random labels; implicit regularization by stochastic gradient descent; flat and sharp minima; double descent in deep networks.

Further reading: UDL chapters 8, 9 and 20 cover measuring performance, regularization, and why deep networks generalize. DLB chapter 7 is a catalogue of regularizers. ML’s model-selection chapter covers the bias–variance decomposition and double descent for linear models.

6. Convolutional networks {#6-convolutional-networks}

Topics: Convolution and cross-correlation; locality, weight sharing, and translation equivariance; padding, stride, and dilation; channels and 1×1 convolutions; pooling and invariance; receptive fields; the cost of a convolutional layer; depthwise-separable convolutions.

Further reading: UDL chapter 10; DLB chapter 9; the CS231n notes on convolutional networks.

7. Convolutional architectures and transfer learning {#7-convolutional-architectures-and-transfer-learning}

Topics: LeNet, AlexNet, VGG, GoogLeNet, residual networks, MobileNet, EfficientNet, and ConvNeXt; design principles and compute–accuracy trade-offs; pretraining and fine-tuning; feature extraction and linear probes.

Further reading: UDL chapter 10 (later sections) and chapter 11.

8. Recurrent networks {#8-recurrent-networks}

Topics: Sequence modeling with shared weights across time; backpropagation through time; vanishing and exploding gradients over time steps; gated units (LSTM and GRU); gradient clipping; bidirectional and stacked networks; encoder–decoder models and teacher forcing.

Further reading: DLB chapter 10 is the most complete textbook treatment. Language modeling with recurrent networks is developed in the NLP and LLMs module.

9. Attention and transformers {#9-attention-and-transformers}

Topics: Attention as differentiable lookup; attention in encoder–decoder models; scaled dot-product self-attention; multiple heads; positional information; the transformer block with residual connections and normalization; masking; computational cost; vision transformers.

Further reading: UDL chapter 12 covers transformers for text and images. Tokenization, pretraining, and language models are developed in the NLP and LLMs module.

10. Self-supervised representation learning {#10-self-supervised-representation-learning}

Topics: Representations and how to evaluate them; autoencoders and denoising; pretext tasks; contrastive learning and the InfoNCE objective; collapse and non-contrastive methods; masked image modeling; linear probes and fine-tuning.

Further reading: UDL chapter 14 frames unsupervised learning; Lilian Weng, Contrastive representation learning collects the contrastive objectives. Variational autoencoders are developed in the Generative AI module.

11. Training at scale and efficient inference {#11-training-at-scale-and-efficient-inference}

Topics: Arithmetic intensity and the memory hierarchy; mixed-precision training; memory for parameters, gradients, optimizer state, and activations; activation checkpointing; data, tensor, and pipeline parallelism; sharded optimizer state; quantization, pruning, and distillation for inference.

Further reading: The PyTorch automatic mixed precision guide documents the numerical formats and loss scaling.

12. Practical methodology {#12-practical-methodology}

Topics: Baselines and sanity checks; overfitting a single batch; monitoring losses, activations, and gradients; learning-rate range tests; hyperparameter search and its budget; interpreting learning curves; reproducibility and seeds; experiment tracking.

Further reading: DLB chapter 11 (practical methodology).

13. Graph neural networks — optional {#13-graph-neural-networks-optional}

Topics: Graphs, adjacency, and permutation symmetry; message passing; graph convolutional and attention networks; node, edge, and graph prediction; expressive power and the Weisfeiler–Lehman test; oversmoothing.

Further reading: UDL chapter 13.

14. Detection and segmentation — optional {#14-detection-and-segmentation-optional}

Topics: Dense prediction; bounding-box regression and intersection over union; two-stage and one-stage detectors; anchors and non-maximum suppression; fully convolutional networks, U-Net, and upsampling; instance segmentation.

15. Infinite width and the neural tangent kernel — optional {#15-infinite-width-and-the-neural-tangent-kernel-optional}

Topics: Wide networks at initialization as Gaussian processes; linearization around initialization; the neural tangent kernel and lazy training; what the kernel limit explains and what it misses; feature learning and parameterization.

Further reading: Lilian Weng, Some math behind neural tangent kernel derives the main results. The Gaussian-process chapter of ML supplies the kernel background.

Exercises and local references

Connections to later modules

Tokenization, pretraining, and language models belong in NLP and LLMs; variational autoencoders, adversarial networks, normalizing flows, and diffusion models in Generative AI; deep reinforcement learning in RL; interpretability, robustness, and alignment in Safety and Frontier. Architectures, optimization, normalization, and training practice common to all of them remain in this module.