Reading plan

Work in progress: this part of the notes is still being revised.

Courses

The order below follows the chapters of this module. Each block lists the material to cover and its primary lectures; further reading and exercises provide additional depth. Blocks 1–14 form the main plan; 15–16 are optional extensions. Timing is flexible.

Slide links are the courses’ original PDFs; CS224N videos are the Spring 2024 recordings except where a block names the 2023 recording. Deep Learning already covered backpropagation, recurrent networks, attention, the transformer block, position encodings, self-supervised learning, and training at scale, so CS224N lectures 3, 5 (second half), and 8 and CS336 lectures 5–8 (GPUs, kernels, and parallelism) serve as review. Policy-gradient methods, which preference tuning and reasoning training use, are developed in the RL module; this module states the objectives they optimize.

Topic list

Reference books used below

1. Text, tokens, and tokenization {#1-text-tokens-and-tokenization}

Topics: Characters, Unicode, and bytes; normalization; words, types, and tokens; Zipf’s and Heaps’ laws; word, character, and byte vocabularies; subword tokenization with byte-pair encoding, WordPiece, and the unigram language model; pre-tokenization and special tokens; compression rate; artifacts of tokenization in arithmetic, code, and non-English text; tokenizer-free models.

Further reading: Sennrich, Haddow, and Birch, Neural machine translation of rare words with subword units (2016), introduced BPE for neural models; Kudo, Subword regularization (2018), the unigram tokenizer; Kudo and Richardson, SentencePiece (2018). CS336 assignment 1 implements a byte-level BPE tokenizer.

2. N-gram language models and perplexity {#2-n-gram-language-models-and-perplexity}

Topics: Language models and the chain rule; the Markov assumption; maximum-likelihood estimation; sparsity and smoothing: add-\(k\), backoff, interpolation, and Kneser–Ney; perplexity and cross-entropy; the entropy of English; sampling from a language model; the noisy channel.

Further reading: Eisenstein chapter 6; Chen and Goodman, An empirical study of smoothing techniques for language modeling (1999), the standard comparison that established modified Kneser–Ney.

3. Word embeddings {#3-word-embeddings}

Topics: The distributional hypothesis; term–document and word–context matrices; pointwise mutual information and PPMI; truncated SVD and latent semantic analysis; word2vec skip-gram and CBOW; negative sampling and its equivalence to factorizing shifted PMI; GloVe; subword embeddings; similarity and analogy evaluation; social biases in embeddings; from static to contextual representations.

Further reading: Mikolov et al., Distributed representations of words and phrases (2013); Pennington, Socher, and Manning, GloVe (2014); Levy, Goldberg, and Dagan, Improving distributional similarity with lessons learned from word embeddings (2015), which shows that the hyperparameters matter more than the algorithm. CS224N assignment 1 compares co-occurrence and word2vec vectors.

4. Transformer language models {#4-transformer-language-models}

Topics: Neural language models from feedforward and recurrent networks to transformers; the decoder-only language model: embeddings, blocks, unembedding, and weight tying; next-token loss and teacher forcing; counting parameters, FLOPs, and memory; the architecture and hyperparameter choices of current models; training instabilities and their fixes; mixture-of-experts layers, routing, and load balancing; attention alternatives and long context.

Further reading: Bengio et al., A neural probabilistic language model (2003); Radford et al., Language models are unsupervised multitask learners (GPT-2, 2019); Karpathy, Let’s reproduce GPT-2 (124M), a complete training run with its engineering. FLLM chapter 2. CS336 assignment 1 implements the model, AdamW, and the training loop.

5. Pretraining and transfer {#5-pretraining-and-transfer}

Topics: Pretrain, then adapt; contextual embeddings; generative pretraining followed by fine-tuning; masked language modeling and its refinements; replaced-token detection; span corruption and the text-to-text format; encoder, decoder, and encoder–decoder models compared; fine-tuning for classification; sentence embeddings; probing what pretraining learns.

Further reading: Devlin et al., BERT (2019); Raffel et al., Exploring the limits of transfer learning with a unified text-to-text transformer (T5, 2020), a systematic comparison of objectives and architectures; Clark et al., ELECTRA (2020). CS224N assignment 4 pretrains a small transformer and fine-tunes it.

6. Pretraining data {#6-pretraining-data}

Topics: Sources: web crawls, books, code, and curated collections; extracting text from HTML; language identification; heuristic and model-based quality filters; exact and near-duplicate removal with MinHash and locality-sensitive hashing; data mixtures and how to choose them; repeating data; contamination of evaluations; synthetic data; licensing, consent, and privacy.

Further reading: Li et al., DataComp-LM (2024), a benchmark for data curation that established classifier-based filtering; Lee et al., Deduplicating training data makes language models better (2022); Muennighoff et al., Scaling data-constrained language models (2023). CS336 assignment 4 builds a filtering and deduplication pipeline on Common Crawl.

7. Scaling laws {#7-scaling-laws}

Topics: Power laws in loss; scaling with parameters, data, and compute; compute-optimal allocation; IsoFLOP profiles and parametric fits; training beyond the compute-optimal point for cheaper inference; scaling with limited data; scaling of hyperparameters and μP; predicting downstream performance; emergent abilities and the choice of metric.

Further reading: Kaplan et al., Scaling laws for neural language models (2020); Hoffmann et al., Training compute-optimal large language models (Chinchilla, 2022); Besiroglu et al., Chinchilla scaling: a replication attempt (2024); Schaeffer, Miranda, and Koyejo, Are emergent abilities of large language models a mirage? (2023). CS336 assignment 3 fits scaling laws under a fixed budget of training runs.

8. Decoding and text generation {#8-decoding-and-text-generation}

Topics: Generation as search and as sampling; greedy and beam search; the likelihood trap and degenerate text; temperature, top-\(k\), nucleus, and min-\(p\) sampling; repetition penalties; constrained decoding and structured output; speculative decoding and why it is exact; the cost of generation: prefill and decode, batching, and paged key–value caches.

Further reading: Holtzman et al., The curious case of neural text degeneration (2020); Meister, Cotterell, and Vieira, If beam search is the answer, what was the question? (2020); Leviathan, Kalman, and Matias, Fast inference from transformers via speculative decoding (2023); Kwon et al., Efficient memory management for large language model serving with PagedAttention (2023).

9. In-context learning and prompting {#9-in-context-learning-and-prompting}

Topics: Few-shot prompting and its dependence on scale; zero-shot instructions; sensitivity to format, order, and label balance; calibration; what the demonstrations contribute; in-context learning as implicit Bayesian inference; transformers that implement learning algorithms in their forward pass; induction heads; the data properties that give rise to in-context learning; prompt design and its limits.

Further reading: Brown et al., Language models are few-shot learners (GPT-3, 2020); Min et al., Rethinking the role of demonstrations (2022); Garg et al., What can transformers learn in-context? (2022); von Oswald et al., Transformers learn in-context by gradient descent (2023). FLLM chapter 3.

10. Fine-tuning and parameter-efficient adaptation {#10-fine-tuning-and-parameter-efficient-adaptation}

Topics: Full fine-tuning; supervised instruction tuning and chat templates; instruction data, collected, templated, and synthetic; forgetting; parameter-efficient methods: adapters, prompt and prefix tuning, LoRA, and QLoRA; the low intrinsic dimension of fine-tuning; model merging and task arithmetic; distilling a stronger model.

Further reading: Wei et al., Finetuned language models are zero-shot learners (FLAN, 2022); Zhou et al., LIMA: less is more for alignment (2023); Hu et al., LoRA (2022); Dettmers et al., QLoRA (2023). RLHF book chapters on instruction tuning.

11. Learning from human preferences {#11-learning-from-human-preferences}

Topics: Why imitation is not enough; collecting preferences; the Bradley–Terry reward model; KL-regularized reward maximization and its optimal policy; RLHF with a policy-gradient optimizer; direct preference optimization and its variants; best-of-\(n\) sampling; reward overoptimization; AI feedback and constitutional methods; what preference tuning changes in a model.

Further reading: Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT, 2022); Rafailov et al., Direct preference optimization (2023); Gao, Schulman, and Hilton, Scaling laws for reward model overoptimization (2023); Bai et al., Constitutional AI (2022). RLHF book chapters on reward modeling, regularization, and direct alignment.

12. Reasoning and test-time compute {#12-reasoning-and-test-time-compute}

Topics: Chain-of-thought prompting and why intermediate tokens add computation; self-consistency and majority voting; best-of-\(n\) with verifiers; outcome and process reward models; search over reasoning steps; reinforcement learning from verifiable rewards; test-time scaling; the limits and faithfulness of reasoning traces.

Further reading: Wei et al., Chain-of-thought prompting elicits reasoning (2022); Wang et al., Self-consistency improves chain of thought reasoning (2023); Lightman et al., Let’s verify step by step (2023); Snell et al., Scaling LLM test-time compute optimally (2024); DeepSeek-AI, DeepSeek-R1 (2025). Welleck et al. sections 4–7. CS336 assignment 5 trains a model on mathematics with expert iteration and GRPO.

13. Retrieval, tools, and agents {#13-retrieval-tools-and-agents}

Topics: Sparse retrieval with TF-IDF and BM25; dense retrieval with dual encoders; approximate nearest-neighbor search; measuring retrieval; retrieval-augmented generation; retrieval versus long context; tool use and function calling; agents that plan, act, and observe; benchmarks for agents; failures, including prompt injection.

Further reading: IIR chapters 6 and 11; Karpukhin et al., Dense passage retrieval (2020); Lewis et al., Retrieval-augmented generation for knowledge-intensive NLP tasks (2020); Yao et al., ReAct (2023); Jimenez et al., SWE-bench (2024).

14. Evaluating language models {#14-evaluating-language-models}

Topics: Perplexity and bits per byte; kinds of benchmarks: multiple choice, free generation, and execution; the pass@\(k\) estimator; contamination; human and model judges and their biases; pairwise comparisons and rating systems; the statistics of evaluation: standard errors, clustered questions, and paired comparisons; holistic evaluation; Goodhart’s law and saturated benchmarks.

Further reading: Miller, Adding error bars to evals (2024); Chen et al., Evaluating large language models trained on code (2021), which defines pass@\(k\); Zheng et al., Judging LLM-as-a-judge (2023); Chiang et al., Chatbot Arena (2024); Biderman et al., Lessons from the trenches on reproducible evaluation of language models (2024).

15. Machine translation and multilingual models — optional {#15-machine-translation-and-multilingual-models-optional}

Topics: Translation as a noisy channel; word alignment and IBM Model 1 trained with EM; phrase-based translation in brief; neural machine translation with encoder–decoder transformers; evaluation with BLEU, chrF, and learned metrics; back-translation; multilingual pretraining and cross-lingual transfer; the curse of multilinguality; low-resource languages.

Further reading: Eisenstein chapter 18; Brown et al., The mathematics of statistical machine translation (1993); Papineni et al., BLEU (2002); Conneau et al., Unsupervised cross-lingual representation learning at scale (XLM-R, 2020). CS224N assignment 3 builds a neural translation system with attention.

16. Syntactic parsing — optional {#16-syntactic-parsing-optional}

Topics: Part-of-speech tagging and named-entity recognition as sequence labeling; constituency, context-free grammars, and treebanks; CKY parsing and probabilistic grammars; dependency grammar and Universal Dependencies; transition-based parsing; graph-based parsing with maximum spanning trees; neural parsers; evaluation; syntax inside language models.

Further reading: Eisenstein chapters 7–11; Chen and Manning, A fast and accurate dependency parser using neural networks (2014); Dozat and Manning, Deep biaffine attention for neural dependency parsing (2017); Hewitt and Manning, A structural probe for finding syntax in word representations (2019). CS224N assignment 2 implements a neural transition-based parser.

Exercises and local references

Connections to later modules

Diffusion language models, multimodal models, and the generative models behind images and audio belong in Generative AI. The policy-gradient methods that optimize the objectives of blocks 11–12 belong in RL. Interpretability of language models, evaluations for dangerous capabilities, the failure modes of preference training, and the theory of alignment belong in Safety and Frontier. Tokenization, language modeling, pretraining, scaling, adaptation, reasoning, retrieval, and evaluation remain in this module.