Reading plan

Work in progress: this part of the notes is still being revised.

Courses

The order below combines the courses by topic. Each block contains the material to cover and its primary lectures; further reading and Python examples provide additional depth. Blocks 1–16 form the main plan; 17–18 are optional extensions. Timing is flexible.

CMU slide links are original PDFs. Cornell’s Fall 2018 topic notes are HTML and use Spring 2017 recordings. Topic numbers differ from recording numbers; the video labels below follow the recordings. Python links lead to official scikit-learn examples with downloadable notebooks.

Topic list

Reference books used below

1. Learning problems and prediction {#1-learning-problems-and-prediction}

Topics: Inputs and targets; regression and classification; models versus fitting algorithms; hypothesis classes; loss, population risk and empirical risk; Bayes prediction; training, validation and test data; simple tree rules as an introductory model.

Further reading: ESL §§2.1–2.5 supplies a statistical account of prediction and decision rules. The Foundations terminology chapter is the local terminology reference.

2. Nearest neighbors and dimensionality {#2-nearest-neighbors-and-dimensionality}

Topics: Nearest-neighbor classification and regression; the role of k; distance and feature scaling; local averaging; Bayes error and the 1-NN result; the curse of dimensionality. Search data structures are an optional extension.

Further reading: ESL §2.3 compares least squares and nearest neighbors; §13.3 develops nearest-neighbor classifiers.

3. Perceptron and linear separation {#3-perceptron-and-linear-separation}

Topics: Hyperplanes and intercepts; signed labels; the perceptron update; separability and geometric margin; the mistake-bound proof; what fails without separability. Keep this elementary algorithm in ML before studying multilayer networks in DL.

Further reading: The margin-dependent mistake bound is a useful first complete algorithm analysis; distinguish it from an i.i.d. generalization guarantee.

4. Linear regression and regularization {#4-linear-regression-and-regularization}

Topics: Least squares; normal equations and projection geometry; Gaussian-noise likelihood; gradient-based fitting; ridge as shrinkage and MAP; lasso and elastic-net formulations; feature scaling and choosing regularization strength. Detailed lasso algorithms and regularization paths are supplementary reading.

Further reading: ESL §§3.2 and 3.4 cover least squares and shrinkage methods. Cornell’s ERM note in topic 7 collects ridge, lasso and elastic-net objectives. Use the Foundations linear-algebra chapter for the projection and stable-solve background.

5. Probabilistic estimation and generative classifiers {#5-probabilistic-estimation-and-generative-classifiers}

Topics: Likelihood, MLE and MAP in learning problems; Bayes classification; conditional independence; categorical, multinomial and Gaussian naive Bayes; smoothing; Gaussian discriminant analysis and LDA/QDA; the assumptions behind decision boundaries. Full-covariance LDA/QDA comes from the additional reading, beyond CMU’s Gaussian naive Bayes.

Further reading: ESL §4.3 supplies LDA/QDA. In Mitchell’s classifier chapter, read the Bayes/naive-Bayes portions here and retain the logistic-regression portions for topic 6.

6. Logistic regression and probabilistic prediction {#6-logistic-regression-and-probabilistic-prediction}

Topics: Conditional likelihood; logits, log-loss and gradients; regularized logistic regression; multiclass/softmax extension; generative versus discriminative estimation; predicted probabilities versus decisions and thresholds. Calibration is an explicit reference addition.

Further reading: ESL §4.4 develops logistic regression, including multiclass modeling. Mitchell’s classifier chapter from topic 5 compares naive Bayes and logistic regression. For optimization, revisit the Foundations calculus chapter as needed.

7. Losses, model selection and bias–variance {#7-losses-model-selection-and-biasvariance}

Topics: ERM and regularized ERM; squared, absolute, logistic, hinge and exponential losses; bias–variance decomposition; underfitting and overfitting; cross-validation; early stopping; model and hyperparameter selection. Extend the course treatment to nested validation, preprocessing leakage, imbalanced metrics and calibration.

Further reading: ESL §2.9 and chapter 7 provide the fuller assessment framework. Scikit-learn: common pitfalls and data leakage connects evaluation rules to actual pipelines.

8. Statistical learning theory {#8-statistical-learning-theory}

Topics: Realizable and agnostic PAC learning; finite-class sample complexity; concentration and uniform convergence; shattering, growth functions, VC dimension and Sauer’s lemma; Rademacher complexity; regularization and capacity control; approximation, estimation and optimization error. Work through representative proofs and compute dimensions/bounds for concrete classes.

Further reading: UML chapters 2–6 give a sustained treatment of PAC learning, uniform convergence and VC theory; chapter 26 extends the complexity viewpoint to Rademacher bounds. These supply proofs and details beyond the three lectures. Foundations’ learning-theory chapter remains the compact local reference.

9. Margins, support vector machines and kernels {#9-margins-support-vector-machines-and-kernels}

Topics: Hard and soft margins; hinge loss; constrained and regularized formulations; Lagrange duality and support vectors; positive semidefinite kernels and feature maps; kernelized perceptron and SVMs; kernel width and regularization. Kernel ridge regression is a short extension.

Further reading: ESL chapter 12 develops margin methods. Welling, Kernel Ridge Regression · PDF supplies the kernel-ridge derivation. Cornell video 21, linked under model selection, introduces kernels; continue with videos 22–25 here.

10. Decision and regression trees {#10-decision-and-regression-trees}

Topics: Recursive partitioning; entropy, information gain and Gini impurity; greedy split selection; categorical and continuous features; regression leaves; stopping and pruning; depth, instability and overfitting. Relate the objective to the tree-building algorithm.

Further reading: ESL §9.2 supplies a fuller CART treatment, including pruning. The introductory CMU lecture in topic 1 provides the first tree example.

11. Bagging and random forests {#11-bagging-and-random-forests}

Topics: Bootstrap samples; averaging unstable learners; variance and correlation; bagged trees; feature subsampling; random forests; out-of-bag evaluation; feature importance and its limitations.

Further reading: ESL §8.7 covers bagging and chapter 15 treats random forests. Breiman, Random Forests (2001) · PDF is the optional original paper.

12. AdaBoost and gradient boosting {#12-adaboost-and-gradient-boosting}

Topics: Weak and strong learning; weighted training examples; AdaBoost updates and exponential loss; training-error guarantees and margins; additive models; gradient descent in function space; gradient-boosted regression trees; shrinkage, tree depth and stopping.

Further reading: ESL chapter 10 supplies the statistical/additive-model view. Schapire, The Boosting Approach to Machine Learning: An Overview · PDF develops the theory and margin perspective.

13. Principal components and dimensionality reduction {#13-principal-components-and-dimensionality-reduction}

Topics: Centering; variance maximization and reconstruction error; SVD and PCA scores; retained dimension; scaling versus standardization; principal-component regression; kernel PCA. Treat PCA as a fitted transformation whose parameters come from training data.

Further reading: ESL §14.5.1 gives the PCA formulation and §3.5 discusses regression on derived inputs. The Foundations linear-algebra chapter contains the required SVD background.

14. Clustering {#14-clustering}

Topics: Clustering objectives and the role of distance; k-means and Lloyd iterations; initialization, local optima and k-means++; hierarchical/agglomerative clustering and linkage; cluster number; stability and interpretation.

Further reading: ESL §14.3 gives broader clustering coverage. Keep objective value, cluster stability and agreement with external labels conceptually separate.

15. Gaussian mixtures and expectation maximization {#15-gaussian-mixtures-and-expectation-maximization}

Topics: Latent component labels; mixture likelihood; soft assignments; complete versus observed data; E and M steps; the lower-bound/Jensen derivation; likelihood monotonicity; initialization, singular covariance and model selection; comparison with k-means.

Further reading: ESL §8.5 develops EM, starting with a Gaussian-mixture example. Separate a guarantee of nondecreasing likelihood from a guarantee of a global optimum.

16. Gaussian processes {#16-gaussian-processes}

Topics: Bayesian linear regression as motivation; distributions over functions; covariance kernels; conditioning for prediction; predictive mean and uncertainty; observation noise and hyperparameter fitting. Bayesian optimization is an optional extension.

Further reading: The GPML chapter index links chapter 4 on covariance functions and chapter 5 on hyperparameters.

17. Semi-supervised and active learning — optional {#17-semi-supervised-and-active-learning-optional}

Topics: How unlabeled data can help under additional assumptions; co-training and graph-based methods; querying labels; uncertainty sampling; selection bias and label complexity. This is an extension after the supervised and unsupervised core.

Further reading: Zhu, Semi-Supervised Learning · PDF and Settles, Active Learning Literature Survey · PDF are optional surveys.

18. Further classical methods — optional {#18-further-classical-methods-optional}

Topics: Kernel density estimation and bandwidth; local regression; basis expansions, splines and generalized additive models; efficient nearest-neighbor search. These extend the breadth of the selected courses.

Further reading: ESL chapter 5 covers basis expansions and splines; chapter 6 covers kernel smoothing and density estimation; §9.1 covers generalized additive models. Quant Prep’s nonparametric-regression, density-estimation and evaluation sections are an additional local reference.

Exercises and local references

Connections to later modules

Multilayer networks and deep-learning training belong in DL; general graphical-model inference in AI; sequence and language models in NLP; deep generative models in Generative AI; bandits, MDPs and control in RL. Classical generative classifiers, Gaussian mixtures, EM, the perceptron and generalization theory remain in this module.