Papers
The notes link these works where their ideas arise. This index groups selected references by their mathematical role.
| Work | Idea developed in the module |
|---|---|
| LeCun, Bottou, Bengio, and Haffner, Gradient-Based Learning Applied to Document Recognition | Convolutional networks for handwritten digits, and a comparison of linear, nearest-neighbor, kernel, and network classifiers on MNIST. |
| DeCoste and Schölkopf, Training Invariant Support Vector Machines | Building knowledge of image transformations into a kernel classifier. |
| Cireşan, Meier, and Schmidhuber, Multi-column Deep Neural Networks for Image Classification | Committees of convolutional networks trained on distorted digits. |
| Krizhevsky, Sutskever, and Hinton, ImageNet Classification with Deep Convolutional Neural Networks | Learned features outperforming hand-designed image features on a large benchmark. |
| Xie, Girshick, and Farhadi, Unsupervised Deep Embedding for Clustering Analysis | Clustering accuracy of k-means on raw MNIST pixels, compared with clustering in a learned representation. |
| Azzalini and Bowman, A Look at Some Data on the Old Faithful Geyser | The geyser’s waiting times as an example of density and conditional density estimation. |
| Baydin, Pearlmutter, Radul, and Siskind, Automatic Differentiation in Machine Learning: a Survey | Forward and reverse differentiation, derivative products, and computational cost. |
| Pearlmutter, Fast Exact Multiplication by the Hessian | Hessian–vector products from forward-mode differentiation of a gradient. |
| Franceschi, Donini, Frasconi, and Pontil, Forward and Reverse Gradient-Based Hyperparameter Optimization | Forward-mode derivatives with respect to a few hyperparameters of a long training run. |
| Duchi, Hazan, and Singer, Adaptive Subgradient Methods for Online Learning and Stochastic Optimization | Accumulated coordinate scaling and adaptive optimization. |
| Kingma and Ba, Adam: A Method for Stochastic Optimization | Moving gradient moments and bias correction. |
| Goh, Why Momentum Really Works | Gradient descent and momentum on quadratics, analyzed one eigendirection at a time; the source of four figures in the calculus chapter. |
| Shannon, A Mathematical Theory of Communication | Entropy, coding, and communication. |
| Valiant, A Theory of the Learnable | A computational formulation of learnability. |
| Vapnik and Chervonenkis, On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities | Uniform convergence and class capacity. |
| Bartlett and Mendelson, Rademacher and Gaussian Complexities: Risk Bounds and Structural Results | Complexity measures for generalization. |
| Bousquet and Elisseeff, Stability and Generalization | How the sensitivity of an algorithm controls generalization. |
| Seeger, PAC-Bayesian Generalisation Error Bounds for Gaussian Process Classification | Generalization bounds for distributions over predictors. |
| Blanchard, Higham, and Higham, Accurate Computation of the Log-Sum-Exp and Softmax Functions | Rounding-error analysis of stable probability calculations. |