Foundations supplies the language, mathematics, algorithms, and numerical tools used throughout machine learning. Its six chapters connect a mathematical description of a learning problem to the computations used to fit and evaluate a model.
Chapters
| Chapter | Main content | Connection to the rest of the module |
|---|---|---|
| 1. Terminology and Mathematical Language | AI, ML, and deep learning; learning paradigms and how RL differs from supervised and unsupervised learning; tasks, including density estimation; parametric and nonparametric models and learning algorithms; prediction, inference, and action; objectives and data splits. | Establishes the meaning of the objects used in the mathematical chapters. |
| 2. Linear Algebra | Vector spaces and linear maps; norms and projections; least squares; eigenvalues and SVD; conditioning; random projections; tensors and tensor derivatives. | Describes representations, parameter arrays, and the geometry of numerical problems. |
| 3. Calculus and Optimization | Derivatives and local approximation; forward and reverse autodiff and when to use each; backpropagation; convexity and smoothness; descent methods and convergence; stochastic and adaptive updates. | Explains what a gradient means, how it is computed, and what an optimization step can guarantee. |
| 4. Probability and Statistics | Probability spaces and random variables, counting, and conditioning; distribution families and transformations of random variables; expectations, limit theorems, and concentration; sampling and estimation; likelihood; uncertainty intervals and the bootstrap; testing; Bayesian inference; decision theory and regression inference. | Distinguishes a population, its observed sample, a fitted model, and uncertainty about each. |
| 5. Information and Learning Theory | Entropy and coding; cross-entropy and KL; mutual information; entropy rates and perplexity; risk, generalization, PAC learning, capacity, margins, and stability. | Explains probabilistic objectives and the conditions under which fitting a sample supports claims about new observations. |
| 6. Numerical Computing with NumPy and PyTorch | Array shapes and storage; broadcasting and contractions; numerical linear algebra; tensors and autograd; parameterized modules, losses, and a complete fitted predictor. | Implements the preceding ideas with explicit shapes, gradients, and evaluation semantics. |
Additional proofs and specialized results appear in collapsed appendices. Chapters 2 and 3 end with an appendix listing the NumPy and PyTorch functions their examples use, each linked to its documentation.
Shared conventions
- An individual vector is a column. A data matrix stores observations in rows, so a linear map \(x\mapsto Wx+b\) becomes \(XW^\top+\mathbf1b^\top\) for a batch. Array shape is stated separately from its mathematical interpretation.
- Derivatives use numerator layout: for \(f:\mathbb R^d\to\mathbb R^m\), the Jacobian \(J_f\) has shape \(m\times d\). A scalar derivative is a row, its Euclidean gradient is a column, and a matrix parameter gradient has the parameter’s shape.
- \(D\) denotes a dataset. \(P_\ast\) denotes its underlying population law in the probability and learning-theory chapters; the terminology chapter also uses \(\mathcal D\) for that law. Sample size is written \(n\) or \(N\) where defined; \(B\) denotes a batch size in the computing chapter.
- A mathematical prediction loss is written with the target first, \(\ell(y,\widehat y)\). A loss viewed as a function of a rule or its parameters is defined locally. Library argument order is specified where it differs.
- Information and Learning Theory measures information in bits, with \(\log=\log_2\), and uses \(\ln\) and nats only where the constant \(e\) is intrinsic to a result. The other chapters, like NumPy and PyTorch, use natural logarithms, so log losses there are in nats. A displayed convergence or generalization guarantee includes the assumptions on the objective, observations, or hypothesis class that make it valid.
Examples and supporting resources
The chapter text contains the definitions, derivations, and worked examples. The computing setup records the Python environment and notebook commands. Inline numerical examples can be run independently. The final chapter also has a companion notebook containing its examples and explanations.
Books and documentation collects the reference texts and official library documentation. Papers collects foundational and supplementary research references. Figure sources records the sources and mathematical models behind the illustrations; the figures added to chapter 1 on 25 September 2026 and to chapters 2, 3, and 4 on 26 September 2026 are regenerated by the code in Sources/Figure code. Course links retains the ML Mastery webpage.
The course PDFs contain worked problem solutions; their notebooks contain solved numerical implementations. The copies below are stored inside this vault, so the links travel with the notes. They follow the course webpage’s stable F01–F08 IDs; their grouping differs from the six chapters above. Companion sources records the source snapshot.
| Supplementary course topic | Notes | Notebook |
|---|---|---|
| Terminology | Notebook | |
| Linear algebra | Notebook | |
| Calculus and optimization | Notebook | |
| Probability | Notebook | |
| Statistical inference | Notebook | |
| Information theory | Notebook | |
| NumPy | Notebook | |
| PyTorch | Notebook |
Connections to later modules
Machine learning develops model families, feature construction, regularization, model selection, and comparative evaluation. Foundations provides the linear algebra, statistical language, and learning guarantees used to analyze those choices. The ML overview lists its chapters.
Deep learning develops architectures and their training behavior, including initialization, normalization, and larger-scale optimization. Foundations provides tensor operations, differentiation, elementary modules, and the meaning of a training update. The DL overview lists its chapters.
Artificial intelligence develops search, logic, probabilistic graphical models, decisions, and causal inference. Foundations provides the probability, conditional independence, Monte Carlo estimation, and information quantities these methods rely on. The AI overview lists its chapters.
NLP and generative modeling develop representations and distributions for language and other structured observations. Foundations provides the probability chain rule, conditional prediction, information quantities, and the numerical form of log-loss objectives.
Reinforcement learning develops Markov chains and Markov decision processes fully, followed by value functions, Bellman equations, planning, and policy learning. The short stochastic-matrix and entropy-rate examples in Foundations supply local context for their own mathematical arguments.