Reading plan

Work in progress: this part of the notes is still being revised.

Courses

The order below follows the chapters of this module. Each block lists the material to cover and its primary lectures; further reading and exercises provide additional depth. Blocks 1–13 form the main plan; 14–15 are optional extensions. Timing is flexible.

Slide links are the courses’ original files; four CS236 lectures are available only as PowerPoint files. Several prerequisites come from earlier modules: the evidence lower bound, Markov chain Monte Carlo, and Langevin-type samplers from AI chapter 10; mixtures, EM, and kernel density estimates from ML; autoencoders, contrastive image–text models, U-Nets, and vision transformers from DL; and autoregressive transformers, tokenization, sampling, and preference optimization from NLP and LLMs. The corresponding introductory lectures serve as review.

Topic list

Reference books used below

1. Foundations of generative modeling {#1-foundations-of-generative-modeling}

Topics: What a generative model is for: density estimation, sampling, and representation; the data manifold and the curse of dimensionality; maximum likelihood as minimizing the forward KL divergence; forward and reverse KL, mode covering and mode seeking; f-divergences and integral probability metrics, including maximum mean discrepancy and the Wasserstein distance; explicit, approximate, and implicit models; the trade-offs among sample quality, diversity, likelihood, and speed.

Further reading: CS236 notes, introduction; Goodfellow, NIPS 2016 tutorial: generative adversarial networks, section 2, for a taxonomy of models by how they use likelihood; Bond-Taylor et al., Deep generative modelling: a comparative review (2022).

2. Autoregressive models {#2-autoregressive-models}

Topics: The chain rule for images and audio; fully visible belief networks and NADE; masked autoencoders for distribution estimation (MADE); PixelRNN and PixelCNN, masked convolutions and the blind spot, gated and conditional PixelCNN; discretized logistic mixtures; WaveNet and dilated causal convolutions; transformers over pixels; orderings; exact likelihood in bits per dimension; the cost of sequential sampling.

Further reading: Germain et al., MADE (2015); van den Oord, Kalchbrenner, and Kavukcuoglu, Pixel recurrent neural networks (2016); van den Oord et al., WaveNet (2016); Salimans et al., PixelCNN++ (2017). CS294-158 homework 1 implements autoregressive models of images.

3. Variational autoencoders {#3-variational-autoencoders}

Topics: Latent-variable models and the intractable marginal likelihood; the evidence lower bound and amortized inference; the reparameterization trick and its variance; Gaussian and Bernoulli decoders; posterior collapse and KL annealing; the rate–distortion view and β-VAE; importance-weighted bounds; conditional VAEs; hierarchical VAEs.

Further reading: Kingma and Welling, An introduction to variational autoencoders (2019); Burda, Grosse, and Salakhutdinov, Importance weighted autoencoders (2016); Alemi et al., Fixing a broken ELBO (2018); Vahdat and Kautz, NVAE (2020). CS294-158 homework 2 trains VAEs on images.

4. Normalizing flows {#4-normalizing-flows}

Topics: The change-of-variables formula; triangular Jacobians; coupling layers in NICE, RealNVP, and Glow; autoregressive flows, MAF and IAF, and their duality; spline flows; residual and continuous-time flows, neural ODEs, and trace estimation; dequantization; what invertibility costs.

Further reading: Papamakarios et al., Normalizing flows for probabilistic modeling and inference (2021); Dinh, Sohl-Dickstein, and Bengio, Density estimation using Real NVP (2017); Chen et al., Neural ordinary differential equations (2018); Lilian Weng, Flow-based deep generative models.

5. Generative adversarial networks {#5-generative-adversarial-networks}

Topics: The minimax game and the optimal discriminator; the Jensen–Shannon divergence; the non-saturating loss; training dynamics, oscillation, and mode collapse; f-GANs; Wasserstein GANs, weight clipping, and gradient penalties; spectral normalization; DCGAN, progressive growing, StyleGAN, and BigGAN; conditional and image-to-image GANs; GAN inversion; adversarial losses inside other models.

Further reading: Goodfellow et al., Generative adversarial nets (2014); Arjovsky, Chintala, and Bottou, Wasserstein GAN (2017); Mescheder, Geiger, and Nowozin, Which training methods for GANs do actually converge? (2018); Karras, Laine, and Aila, StyleGAN (2019). CS294-158 homework 3 trains adversarial networks.

6. Energy-based models and score matching {#6-energy-based-models-and-score-matching}

Topics: Energy functions and the partition function; the maximum-likelihood gradient and its negative phase; Langevin dynamics and contrastive divergence; noise-contrastive estimation; the score function and the Fisher divergence; score matching, sliced score matching, and denoising score matching; Tweedie’s formula; the failure of Langevin sampling in low-density regions; noise-conditional score networks and annealed Langevin dynamics.

Further reading: Song and Kingma, How to train your energy-based models (2021); Hyvärinen, Estimation of non-normalized statistical models by score matching (2005); Vincent, A connection between score matching and denoising autoencoders (2011); Song and Ermon, Generative modeling by estimating gradients of the data distribution (2019).

7. Denoising diffusion models {#7-denoising-diffusion-models}

Topics: The forward noising process and its closed form; the reverse process as a hierarchical latent-variable model; the evidence lower bound and its decomposition; predicting the noise, the data, or the velocity; the simplified loss and its link to denoising score matching; noise schedules and signal-to-noise weighting; ancestral sampling; DDIM and deterministic sampling; U-Net denoisers with time conditioning.

Further reading: Ho, Jain, and Abbeel, Denoising diffusion probabilistic models (2020); Song, Meng, and Ermon, Denoising diffusion implicit models (2021); Kingma et al., Variational diffusion models (2021); Luo, Understanding diffusion models: a unified perspective (2022); Lilian Weng, What are diffusion models?. CS294-158 homework 4 trains a diffusion model.

8. Diffusion SDEs and fast sampling {#8-diffusion-sdes-and-fast-sampling}

Topics: Stochastic differential equations and their Euler–Maruyama discretization; variance-preserving and variance-exploding processes; the Fokker–Planck equation; reverse-time SDEs; the probability-flow ODE and exact likelihoods; numerical solvers, Heun’s method, and exponential integrators; the design space of noise levels, preconditioning, and time steps; distillation, consistency models, and few-step generation.

Further reading: Song et al., Score-based generative modeling through stochastic differential equations (2021); Karras et al., Elucidating the design space of diffusion-based generative models (2022); Lu et al., DPM-Solver (2022); Song et al., Consistency models (2023).

9. Flow matching {#9-flow-matching}

Topics: Flows as solutions of ordinary differential equations; the continuity equation; probability paths and conditional paths; the conditional flow matching loss and why it has the same gradient as the marginal one; Gaussian paths and their relation to diffusion; rectified flows and straightening; stochastic interpolants; couplings from optimal transport; sampling with few steps.

Further reading: Lipman et al., Flow matching for generative modeling (2023); Liu, Gong, and Liu, Flow straight and fast (2023); Albergo and Vanden-Eijnden, Building normalizing flows with stochastic interpolants (2023); Tong et al., Improving and generalizing flow-based generative models with minibatch optimal transport (2024). The 6.S184 labs implement flow matching and score matching from scratch.

10. Guidance and conditional generation {#10-guidance-and-conditional-generation}

Topics: Conditional generative models; classifier guidance; classifier-free guidance and its geometry; the trade-off between fidelity and diversity; negative prompts; solving inverse problems with a diffusion prior: inpainting, super-resolution, and posterior sampling; editing by partial noising and inversion; adding control signals; fine-tuning for personalization; aligning generators with rewards and preferences.

Further reading: Dhariwal and Nichol, Diffusion models beat GANs on image synthesis (2021); Ho and Salimans, Classifier-free diffusion guidance (2022); Chung et al., Diffusion posterior sampling for general noisy inverse problems (2023); Zhang, Rao, and Agrawala, Adding conditional control to text-to-image diffusion models (2023).

11. Latent diffusion and large-scale generation {#11-latent-diffusion-and-large-scale-generation}

Topics: Why generate in a latent space; autoencoders for latent diffusion and their losses; text conditioning with frozen text encoders and cross-attention; U-Nets and diffusion transformers; rectified-flow transformers at scale; resolution, noise schedules, and data; video generation with spatiotemporal latents; the cost of training and sampling; distillation for deployment.

Further reading: Rombach et al., High-resolution image synthesis with latent diffusion models (2022); Peebles and Xie, Scalable diffusion models with transformers (2023); Esser et al., Scaling rectified flow transformers for high-resolution image synthesis (2024); Sander Dieleman, Generative modelling in latent space.

12. Discrete tokens and multimodal generation {#12-discrete-tokens-and-multimodal-generation}

Topics: Discrete latent variables, the Gumbel-softmax relaxation, and the straight-through estimator; vector quantization, VQ-VAE, and VQGAN; residual and finite scalar quantization; neural audio codecs; autoregressive transformers over image and audio tokens; masked generative transformers; discrete diffusion, masked diffusion, and diffusion language models; vision–language models and unified models that generate several modalities.

Further reading: van den Oord, Vinyals, and Kavukcuoglu, Neural discrete representation learning (2017); Esser, Rombach, and Ommer, Taming transformers for high-resolution image synthesis (2021); Chang et al., MaskGIT (2022); Sahoo et al., Simple and effective masked diffusion language models (2024); Sander Dieleman, Diffusion language models.

13. Evaluating generative models {#13-evaluating-generative-models}

Topics: Likelihood and bits per dimension, and why good likelihoods and good samples can disagree; the Inception score; the Fréchet inception distance, its bias, and the choice of feature space; kernel inception distance; precision and recall for distributions; text–image alignment scores; human preference studies; memorization, data copying, and extracting training data.

Further reading: Theis, van den Oord, and Bethge, A note on the evaluation of generative models (2016); Heusel et al., GANs trained by a two time-scale update rule converge to a local Nash equilibrium (2017), which introduced FID; Kynkäänniemi et al., Improved precision and recall metric for assessing generative models (2019); Stein et al., Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models (2023); Carlini et al., Extracting training data from diffusion models (2023).

14. Generative models for science and control — optional {#14-generative-models-for-science-and-control-optional}

Topics: Generative models as priors and samplers outside media: diffusion policies for robot control and planning; protein structure generation and design; molecules and docking; weather and climate ensembles; equivariance to rotations and translations; conditioning on physical constraints; how such models are validated.

Further reading: Chi et al., Diffusion policy (2023); Janner et al., Planning with diffusion for flexible behavior synthesis (2022); Watson et al., De novo design of protein structure and function with RFdiffusion (2023); Price et al., Probabilistic weather forecasting with machine learning (2025).

15. The theory of diffusion models — optional {#15-the-theory-of-diffusion-models-optional}

Topics: Convergence of diffusion sampling given an accurate score; the role of score error and discretization; why the optimal denoiser memorizes the training set and why trained networks generalize; inductive biases of convolutional denoisers; diffusion as spectral autoregression; Schrödinger bridges and connections to optimal transport.

Further reading: Chen et al., Sampling is as easy as learning the score (2023); Kadkhodaie et al., Generalization in diffusion models arises from geometry-adaptive harmonic representations (2024); Kamb and Ganguli, An analytic theory of creativity in convolutional diffusion models (2025); De Bortoli et al., Diffusion Schrödinger bridge with applications to score-based generative modeling (2021).

Exercises and local references

Connections to later modules

Reinforcement learning uses generative models as policies and world models, and fine-tuning a diffusion model against a reward is a reinforcement-learning problem developed in RL. Watermarking, provenance, deepfakes, the misuse of generators, and the memorization of copyrighted or private data are taken up in Safety and Frontier. The families of generative models, their training objectives, samplers, guidance, architectures, and evaluation remain in this module.