Books and documentation
The chapter notes develop the module’s main material. These books and official documentation provide alternative explanations, fuller proofs, and the details of the library interfaces used in the code. The reading plan lists the lecture slides, course notes, and scikit-learn examples for each topic.
Main references
| Source | Relevant material |
|---|---|
| Hastie, Tibshirani, and Friedman, The Elements of Statistical Learning, 2nd edition | The statistical reference for most chapters: §§2.3–2.9 (chapters 1 and 6), 3 (chapter 3), 4 (chapters 4–5), 5–6 and §9.1 (chapter 17), 7 (chapter 6), §8.5 (chapter 14), §8.7 and 15 (chapter 10), §9.2 (chapter 9), 10 (chapter 11), 12 (chapter 8), §13.3 (chapter 1), §§14.3 and 14.5 (chapters 12–13). The authors’ site hosts the PDF. |
| Shalev-Shwartz and Ben-David, Understanding Machine Learning | Chapters 2–6 for PAC learning, uniform convergence, and VC dimension; chapter 7 for structural risk minimization; chapter 21 for online learning; chapter 26 for Rademacher complexity. The proof reference for chapter 7. If the authors’ server is unavailable, the publisher’s page has the full text, which may require library access. |
| Bishop, Pattern Recognition and Machine Learning | Chapter 4 for linear classifiers, chapter 6 for kernels and Gaussian processes, chapter 9 for mixtures and EM, and chapter 12 for probabilistic PCA. |
| Murphy, Probabilistic Machine Learning: An Introduction | A probabilistic account of the same supervised and unsupervised methods, with current notation; also listed in the Foundations references. |
Specialized books
| Source | Relevant material |
|---|---|
| Devroye, Györfi, and Lugosi, A Probabilistic Theory of Pattern Recognition | Consistency of nearest-neighbor and partitioning rules, Stone’s theorem, and distribution-free lower bounds (chapter 1). |
| Györfi, Kohler, Krzyżak, and Walk, A Distribution-Free Theory of Nonparametric Regression | Rates of convergence for local averaging, nearest-neighbor, and partitioning estimates (chapters 1 and 9). |
| Kearns and Vazirani, An Introduction to Computational Learning Theory | PAC learning of concrete classes, Occam’s razor, and computational hardness of learning (chapter 7). |
| Minsky and Papert, Perceptrons | What single-layer perceptrons cannot represent (chapter 2). |
| Schölkopf and Smola, Learning with Kernels | Kernels, reproducing kernel Hilbert spaces, support vector machines, and kernel PCA (chapters 8 and 12). |
| Breiman, Friedman, Olshen, and Stone, Classification and Regression Trees | The CART algorithm, surrogate splits, and cost-complexity pruning (chapter 9). |
| Quinlan, C4.5: Programs for Machine Learning | Information-gain trees, gain ratio, and rule extraction (chapter 9). |
| Schapire and Freund, Boosting: Foundations and Algorithms | AdaBoost, its training and generalization analyses, margins, and game-theoretic views; open access (chapter 11). |
| Rasmussen and Williams, Gaussian Processes for Machine Learning | Chapter 2 for regression, chapter 4 for covariance functions, chapter 5 for the marginal likelihood, and chapter 6 for the relation to kernel methods and splines; free chapters (chapter 15). |
| Chapelle, Schölkopf, and Zien (eds.), Semi-Supervised Learning | Assumptions, generative, low-density, and graph-based methods, and benchmark comparisons (chapter 16). |
| Silverman, Density Estimation for Statistics and Data Analysis | Kernel density estimation and bandwidth selection (chapter 17). |
| Neal, Bayesian Learning for Neural Networks | The limit in which a wide Bayesian neural network becomes a Gaussian process (chapter 15). |
Notes and guides
| Source | Relevant material |
|---|---|
| Mitchell, Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression | A book-chapter draft comparing the two estimators (chapters 4–5). |
| Welling, Kernel Ridge Regression | A short derivation of the dual solution (chapter 8). |
| Hsu, Chang, and Lin, A Practical Guide to Support Vector Classification | Scaling, kernel choice, and grid search over \((C,\gamma)\) (chapter 8). |
Library documentation
The links point to the current stable documentation. The code in the notes was tested with scikit-learn 1.4.2, whose documentation is archived at scikit-learn.org/1.4; defaults and parameter names occasionally change between versions.
| Documentation | Relevant material |
|---|---|
| scikit-learn nearest neighbors and density estimation | \(k\)-NN classifiers and regressors, \(k\)-d trees and ball trees, and KernelDensity (chapters 1 and 17). |
| scikit-learn linear models | Least squares, Ridge, Lasso, ElasticNet, and LogisticRegression, including their penalty parameterizations (chapters 3 and 5). |
| scikit-learn naive Bayes and discriminant analysis | Event models, smoothing, LDA, QDA, and shrinkage covariance estimates (chapter 4). |
| scikit-learn probability calibration | Reliability diagrams, Platt scaling, and isotonic regression (chapter 5). |
| scikit-learn cross-validation, model evaluation, and common pitfalls | Splitters, scoring rules, metrics, pipelines, and data leakage (chapter 6). |
| scikit-learn support vector machines and kernel approximation | SVC, LinearSVC, kernel parameters, multiclass strategies, Nyström and random Fourier features (chapter 8). |
| LIBSVM | The solver behind scikit-learn’s SVC (chapter 8). |
| scikit-learn decision trees | Splitting criteria, missing values, and cost-complexity pruning (chapter 9). |
| scikit-learn ensembles | Bagging, random forests, extremely randomized trees, AdaBoost, and gradient boosting, including histogram-based boosting (chapters 10–11). |
| scikit-learn permutation importance and partial dependence | Model inspection for forests and boosted trees (chapters 10–11). |
| scikit-learn matrix decompositions | PCA, probabilistic PCA scores, and kernel PCA (chapter 12). |
| scikit-learn clustering | \(k\)-means, agglomerative clustering, DBSCAN, spectral clustering, and clustering metrics (chapter 13). |
| scikit-learn Gaussian mixtures | EM for mixtures, covariance types, initialization, and model selection (chapter 14). |
| scikit-learn Gaussian processes | Kernels, hyperparameter fitting, and GaussianProcessRegressor (chapter 15). |
| scikit-learn semi-supervised learning | Self-training, label propagation, and label spreading (chapter 16). |
SciPy make_smoothing_spline and scikit-learn SplineTransformer | Smoothing splines with generalized cross-validation, and B-spline features for linear models (chapter 17). |