This course introduces generative AI through the study of large language models, combining mathematical foundations with hands-on implementation. Over the eight weeks we build a small language model from scratch, small enough to train on a laptop in minutes to an hour: we implement its components, train and fine-tune it, probe its behavior, and evaluate its outputs. The aim is a complete working understanding of the model: how it works, how to build it, and how to investigate what it has learned.
We trace the path from statistical models of language to Transformers, pretraining, post-training, and inference. Throughout, we ask the same questions: what can these models learn and compute, from what data, with what resources, and with what guarantees? The implementation gives a concrete setting in which to explore these questions, connecting theory with experiment.
How these materials are made. The course materials are put together with the help of AI. I do the high-level planning, and I read and refine all of the material thoroughly myself. For this reason the materials are constantly updated and refined. This is the first version of the course, so please bear with me :)
What does it mean for a machine to be intelligent, to learn, or to use language? We begin with a short history, from the first models of computation to today’s language models, and set out what the course will build. After fixing the vocabulary of AI, machine learning, and deep learning, we look at how intelligence, learning, language, and meaning have been approached: Turing’s imitation game, search and learning in game playing, induction and PAC learning, grammars and distributional evidence, and the relationship between form, meaning, and intent. (notes, slides, slides as PowerPoint)
Hands-on: a tensor-based autograd engine built from scratch, covered during the lecture's hands-on segment: computation graphs, reverse-mode differentiation, and gradient checks. We use it to train logistic regression and a two-layer network—the first piece of the codebase that grows into our language model. (notebook)
Two ideas older than neural networks run through this lecture: text as the output of a random source (Markov, Shannon), and words as points in a space. We build language models by counting: n-gram models as maximum likelihood estimates, perplexity, and smoothing. We then build word embeddings, the idea behind semantic search and recommendation, by counting contexts and by prediction (word2vec), and see why the two agree. Finally we build neural language models step by step: the bigram model as a network, the fixed-window model of Bengio et al., and recurrent networks, with backpropagation through time, vanishing gradients, and the additive path of the LSTM and residual networks. We end with what recurrence leaves unsolved, the starting point of Week 3. (notes, slides, slides as PowerPoint)
Hands-on: in class, four short stops: counting and Shannon’s samples; word vectors from counts and from word2vec; the bigram network and the fixed-window model; recurrent networks, samples, and a memory test (in-class notebook). To take home, every experiment of the notes, including smoothing, word2vec trained live, and character- and word-level LSTMs (notebook).
A language model predicts what comes next. A translator must also carry what a source sentence says into a target sentence, despite changes in length and word order. Early neural translators encoded the source into a fixed-size state from which a decoder generated the translation, so every word was predicted from the same summary, and some lost accuracy as sentences grew longer. Attention gave the decoder a fresh, learned read of the source at each step. A separate difficulty remained: recurrent networks compute one position at a time, even in training. Convolution and then self-attention removed this chain, leading to the Transformer’s combination of direct access to source information and parallel computation within each layer during training. (notes, slides, slides as PowerPoint)
Hands-on: writing the attention step of a trained English–French translator and checking it against the model; comparing the translator with and without attention; and seeing where it looks as it translates, including at a word it gets wrong. (notebook)
Background Notes on Machine Learning and AI
Updated 14 October 2026.