Books and documentation
The chapter notes develop the module’s main material. These books and documentation pages provide alternative explanations, fuller treatments of the classical methods, and the tools used for real work with language models. The reading plan lists the lecture slides and readings for each topic.
Main references
| Source | Relevant material |
|---|---|
| Jurafsky and Martin, Speech and Language Processing, third edition draft of August 19, 2026 | The standard textbook, free as PDFs by chapter with slides: chapter 2 (chapter 1), 3 and appendix C (chapter 2), 5 and appendix J (chapter 3), 7 (chapter 4), 9 (chapter 5), 11 (chapter 13), 13 (chapter 15), and 18–20 with appendix E (chapter 16). Chapter numbers change between drafts. |
| Eisenstein, Introduction to Natural Language Processing (MIT Press, 2019) | Free draft PDF, stronger on the classical methods: chapter 6 (chapter 2), 14 (chapter 3), 18 (chapter 15), and 7–11 (chapter 16). |
| Xiao and Zhu, Foundations of Large Language Models (2025) | Free; pretraining, generative models, prompting, alignment, and inference (chapters 4–12). |
| Lambert, Reinforcement Learning from Human Feedback | Free; instruction tuning, reward models, policy-gradient and direct preference optimization, and post-training recipes (chapters 10–12). |
| Manning, Raghavan, and Schütze, Introduction to Information Retrieval (Cambridge, 2008) | Free; inverted indexes, tf-idf weighting, probabilistic retrieval, and evaluation (chapter 13). |
Specialized books and surveys
| Source | Relevant material |
|---|---|
| Raschka, Build a Large Language Model (From Scratch) (Manning, 2024), with code | A GPT built step by step in PyTorch: tokenization, attention, pretraining, and fine-tuning for classification and instructions (chapters 1, 4, and 10). |
| Tunstall, von Werra, and Wolf, Natural Language Processing with Transformers (O’Reilly, 2022), with notebooks | Practical use of the Hugging Face libraries for classification, named-entity recognition, generation, summarization, and question answering (chapters 5, 8, 10, and 13). |
| Alammar and Grootendorst, Hands-On Large Language Models (O’Reilly, 2024), with code | Illustrated treatment of tokens and embeddings, prompting, retrieval-augmented generation, and fine-tuning (chapters 1, 5, 9–10, and 13). |
| Welleck et al., From Decoding to Meta-Generation: Inference-Time Algorithms for Large Language Models (2024) | A survey of token-level decoding, sampling and search over whole outputs, and efficient generation (chapters 8 and 12). |
| Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009) | The derivation of BM25 and its extensions (chapter 13). |
| Koehn, Statistical Machine Translation (Cambridge, 2010), and Neural Machine Translation (Cambridge, 2020) | Word alignment, phrase-based models, and decoding; then neural encoder–decoder translation, its training, and its evaluation (chapter 15). |
| Kübler, McDonald, and Nivre, Dependency Parsing (Morgan & Claypool, 2009) | Transition-based and graph-based dependency parsing in depth (chapter 16). |
Library documentation
| Source | Relevant material |
|---|---|
| PyTorch documentation | nn.TransformerEncoderLayer for the character transformer (chapters 4, 5, 8, and 10), torch.nn.utils.parametrize for LoRA (chapter 10), and optimizers and schedulers. |
| NumPy documentation | Arrays and random generation; used in almost every chapter. |
| SciPy documentation | kmeans2 for inverted files and product quantization (chapter 13), logsumexp (chapter 16), and minimize for fitting scaling laws (chapter 7). |
| scikit-learn t-SNE | The embedding map of chapter 3. |
Tools for larger problems
The notes implement every method from scratch on small examples. For real models and data, mature implementations exist:
| Tool | Use |
|---|---|
| Hugging Face Transformers, Tokenizers, and Datasets | Pretrained models, fast tokenizers, and datasets (chapters 1, 4–5, and 10). |
| tiktoken and SentencePiece | Production byte-level BPE and unigram tokenizers (chapter 1). |
| KenLM | Large Kneser–Ney n-gram models (chapter 2). |
| DataTrove | Filtering and deduplication of web-scale text, as used to build FineWeb (chapter 6). |
| vLLM | Efficient serving with paged key–value caches, continuous batching, and speculative decoding (chapter 8). |
| PEFT and TRL | LoRA and other adapters; supervised fine-tuning, reward modeling, DPO, and GRPO (chapters 10–12). |
| FAISS | Approximate nearest-neighbor search with inverted files, product quantization, and graphs (chapter 13). |
| Language Model Evaluation Harness | Standardized benchmark evaluation (chapter 14). |
| SacreBLEU | Reproducible BLEU and chrF scores (chapter 15). |
| Stanza and spaCy | Tokenization, tagging, named entities, and dependency parsing in many languages (chapter 16). |