Applied Machine Learning

Preface......Page 3
Contents......Page 8
Classification......Page 17
1 Learning to Classify......Page 18
1.1.1 The Error Rate and Other Summaries of Performance......Page 19
1.1.2 More Detailed Evaluation......Page 20
1.1.3 Overfitting and Cross-Validation......Page 21
1.2 Classifying with Nearest Neighbors......Page 22
1.2.1 Practical Considerations for Nearest Neighbors......Page 23
1.3 Naive Bayes......Page 25
1.3.1 Cross-Validation to Choose a Model......Page 28
1.3.2 Missing Data......Page 30
1.4.2 Remember These Facts......Page 31
1.4.4 Be Able to......Page 32
2.1 The Support Vector Machine......Page 35
2.1.1 The Hinge Loss......Page 36
2.1.2 Regularization......Page 38
2.1.3 Finding a Classifier with Stochastic Gradient Descent......Page 39
2.1.4 Searching for λ......Page 41
2.1.5 Summary: Training with Stochastic Gradient Descent......Page 43
2.1.6 Example: Adult Income with an SVM......Page 44
2.1.7 Multiclass Classification with SVMs......Page 47
2.2 Classifying with Random Forests......Page 48
2.2.1 Building a Decision Tree......Page 49
2.2.2 Choosing a Split with Information Gain......Page 52
2.2.4 Building and Evaluating a Decision Forest......Page 55
2.2.5 Classifying Data Items with a Decision Forest......Page 56
2.3.2 Remember These Facts......Page 58
2.3.4 Be Able to......Page 59
3.1 Held-Out Loss Predicts Test Loss......Page 63
3.1.1 Sample Means and Expectations......Page 64
3.1.3 A Generalization Bound......Page 66
3.2 Test and Training Error for a Classifier from a Finite Family......Page 67
3.2.1 Hoeffding's Inequality......Page 68
3.2.2 Test from Training for a Finite Family of Predictors......Page 69
3.2.3 Number of Examples Required......Page 70
3.3.1 Predictors and Binary Functions......Page 71
3.3.2 Symmetrization......Page 75
3.3.3 Bounding the Generalization Error......Page 76
3.4.2 Remember These Facts......Page 78
3.4.3 Be Able to......Page 79
High Dimensional Data......Page 80
4.1 Summaries and Simple Plots......Page 81
4.1.2 Stem Plots and Scatterplot Matrices......Page 82
4.1.3 Covariance......Page 84
4.1.4 The Covariance Matrix......Page 86
4.2.1 The Curse: Data Isn't Where You Think It Is......Page 89
4.2.2 Minor Banes of Dimension......Page 90
4.3 Using Mean and Covariance to Understand High Dimensional Data......Page 91
4.3.1 Mean and Covariance Under Affine Transformations......Page 92
4.3.2 Eigenvectors and Diagonalization......Page 93
4.3.3 Diagonalizing Covariance by Rotating Blobs......Page 94
4.4 The Multivariate Normal Distribution......Page 95
4.4.1 Affine Transformations and Gaussians......Page 96
4.4.2 Plotting a 2D Gaussian: Covariance Ellipses......Page 97
4.4.3 Descriptive Statistics and Expectations......Page 98
4.4.4 More from the Curse of Dimension......Page 99
4.5.2 Remember These Facts......Page 100
4.5.3 Remember These Procedures......Page 101
5.1.1 Approximating Blobs......Page 104
5.1.2 Example: Transforming the Height–Weight Blob......Page 105
5.1.3 Representing Data on Principal Components......Page 107
5.1.4 The Error in a Low Dimensional Representation......Page 109
5.1.5 Extracting a Few Principal Components with NIPALS......Page 110
5.1.6 Principal Components and Missing Values......Page 112
5.1.7 PCA as Smoothing......Page 114
5.2 Example: Representing Colors with Principal Components......Page 116
5.3 Example: Representing Faces with Principal Components......Page 120
5.4.4 Be Able to......Page 122
6.1 The Singular Value Decomposition......Page 127
6.1.1 SVD and PCA......Page 129
6.1.3 Smoothing with the SVD......Page 130
6.2.1 Choosing Low D Points Using High D Distances......Page 132
6.2.2 Using a Low Rank Approximation to Factor......Page 133
6.2.3 Example: Mapping with Multidimensional Scaling......Page 134
6.3 Example: Text Models and Latent Semantic Analysis......Page 136
6.3.1 The Cosine Distance......Page 137
6.3.2 Smoothing Word Counts......Page 138
6.3.4 Obtaining the Meaning of Words......Page 140
6.3.5 Example: Mapping NIPS Words......Page 143
6.3.6 TF-IDF......Page 144
6.4.4 Be Able to......Page 146
7.1 Canonical Correlation Analysis......Page 149
7.2 Example: CCA of Words and Pictures......Page 152
7.3 Example: CCA of Albedo and Shading......Page 154
7.3.1 Are Correlations Significant?......Page 158
7.4.4 Be Able to......Page 160
Clustering......Page 162
8.1 Agglomerative and Divisive Clustering......Page 163
8.1.1 Clustering and Distance......Page 165
8.2 The k-Means Algorithm and Variants......Page 167
8.2.1 How to Choose k......Page 171
8.2.2 Soft Assignment......Page 172
8.2.3 Efficient Clustering and Hierarchical k-Means......Page 174
8.2.5 Example: Groceries in Portugal......Page 175
8.2.6 General Comments on k-Means......Page 178
8.3 Describing Repetition with Vector Quantization......Page 179
8.3.1 Vector Quantization......Page 180
8.3.2 Example: Activity from Accelerometer Data......Page 183
8.4.3 Remember These Procedures......Page 186
9.1 Mixture Models and Clustering......Page 191
9.1.1 A Finite Mixture of Blobs......Page 192
9.1.2 Topics and Topic Models......Page 193
9.2 The EM Algorithm......Page 196
9.2.1 Example: Mixture of Normals: The E-step......Page 197
9.2.2 Example: Mixture of Normals: The M-step......Page 199
9.2.3 Example: Topic Model: The E-step......Page 200
9.2.5 EM in Practice......Page 201
9.3.4 Be Able to......Page 206
Regression......Page 211
10.1 Overview......Page 212
10.1.1 Regression to Spot Trends......Page 213
10.2 Linear Regression and Least Squares......Page 215
10.2.1 Linear Regression......Page 216
10.2.2 Choosing β......Page 217
10.2.4 R-squared......Page 219
10.2.5 Transforming Variables......Page 221
10.2.6 Can You Trust Your Regression?......Page 224
10.3 Visualizing Regressions to Find Problems......Page 225
10.3.1 Problem Data Points Have Significant Impact......Page 226
10.3.2 The Hat Matrix and Leverage......Page 229
10.3.3 Cook's Distance......Page 230
10.3.4 Standardized Residuals......Page 231
10.4 Many Explanatory Variables......Page 232
10.4.2 Regularizing Linear Regressions......Page 234
10.4.3 Example: Weight Against Body Measurements......Page 239
10.5.3 Remember These Procedures......Page 243
10.5.4 Be Able to......Page 244
11.1 Model Selection: Which Model Is Best?......Page 252
11.1.1 Bias and Variance......Page 253
11.1.2 Choosing a Model Using Penalties: AIC and BIC......Page 255
11.1.3 Choosing a Model Using Cross-Validation......Page 257
11.1.4 Greedy Search with Stagewise Regression......Page 258
11.1.5 What Variables Are Important?......Page 259
11.2 Robust Regression......Page 260
11.2.1 M-Estimators and Iteratively Reweighted Least Squares......Page 261
11.2.2 Scale for M-Estimators......Page 264
11.3.1 Logistic Regression......Page 265
11.3.2 Multiclass Logistic Regression......Page 267
11.3.3 Regressing Count Data......Page 268
11.4 L1 Regularization and Sparse Models......Page 269
11.4.1 Dropping Variables with L1 Regularization......Page 270
11.4.2 Wide Datasets......Page 274
11.4.3 Using Sparsity Penalties with Other Models......Page 277
11.5.2 Remember These Facts......Page 278
11.5.3 Remember These Procedures......Page 279
12 Boosting......Page 282
12.1.1 Example: Greedy Stagewise Linear Regression......Page 283
12.1.3 Greedy Stagewise Regression with Trees......Page 286
12.2.1 The Loss......Page 291
12.2.2 Recipe: Stagewise Reduction of Loss......Page 293
12.2.3 Example: Boosting Decision Stumps......Page 295
12.2.4 Gradient Boost with Decision Stumps......Page 296
12.2.5 Gradient Boost with Other Predictors......Page 297
12.2.6 Example: Is a Prescriber an Opiate Prescriber?......Page 298
12.2.7 Pruning the Boosted Predictor with the Lasso......Page 300
12.2.8 Gradient Boosting Software......Page 301
12.3 You Should......Page 305
12.3.3 Remember These Facts......Page 306
12.3.5 Be Able to......Page 307
Graphical Models......Page 310
13.1 Markov Chains......Page 311
13.1.1 Transition Probability Matrices......Page 315
13.1.2 Stationary Distributions......Page 317
13.1.3 Example: Markov Chain Models of Text......Page 319
13.2.1 Hidden Markov Models......Page 322
13.2.2 Picturing Inference with a Trellis......Page 323
13.2.3 Dynamic Programming for HMMs: Formalities......Page 326
13.2.4 Example: Simple Communication Errors......Page 327
13.3 Learning an HMM......Page 329
13.3.2 Learning an HMM with EM......Page 330
13.4.1 Remember These Terms......Page 335
13.4.3 Be Able to......Page 336
14.1 Graphical Models......Page 339
14.1.1 Inference and Graphs......Page 340
14.1.2 Graphical Models......Page 342
14.1.3 Learning in Graphical Models......Page 343
14.2 Conditional Random Field Models for Sequences......Page 344
14.2.1 MEMMs and Label Bias......Page 345
14.2.2 Conditional Random Field Models......Page 347
14.2.3 Learning a CRF Takes Care......Page 348
14.3.1 Representing the Model......Page 349
14.3.2 Example: Modelling a Sequence of Digits......Page 350
14.3.3 Setting Up the Learning Problem......Page 351
14.3.4 Evaluating the Gradient......Page 352
14.4.3 Be Able to......Page 354
15.1 Useful but Intractable Models......Page 357
15.1.1 Denoising Binary Images with Boltzmann Machines......Page 358
15.1.2 A Discrete Markov Random Field......Page 359
15.1.3 Denoising and Segmenting with Discrete MRFs......Page 360
15.1.4 MAP Inference in Discrete MRFs Can Be Hard......Page 363
15.2 Variational Inference......Page 364
15.2.1 The KL Divergence......Page 365
15.2.2 The Variational Free Energy......Page 366
15.3 Example: Variational Inference for Boltzmann Machines......Page 367
15.4.3 Be Able to......Page 370
Deep Networks......Page 371
16.1 Units and Classification......Page 372
16.1.1 Building a Classifier out of Units: The Cost Function......Page 373
16.1.2 Building a Classifier out of Units: Strategy......Page 374
16.1.3 Building a Classifier out of Units: Training......Page 375
16.2 Example: Classifying Credit Card Accounts......Page 377
16.3.1 Stacking Layers......Page 382
16.3.2 Jacobians and the Gradient......Page 384
16.3.3 Setting up Multiple Layers......Page 385
16.3.4 Gradients and Backpropagation......Page 386
16.4 Training Multilayer Networks......Page 388
16.4.1 Software Environments......Page 390
16.4.2 Dropout and Redundant Units......Page 391
16.4.3 Example: Credit Card Accounts Revisited......Page 392
16.4.4 Advanced Tricks: Gradient Scaling......Page 395
16.5.3 Remember These Procedures......Page 399
16.5.4 Be Able to......Page 400
17.1 Image Classification......Page 404
17.1.1 Pattern Detection by Convolution......Page 406
17.1.2 Convolutional Layers upon Convolutional Layers......Page 412
17.2 Two Practical Image Classifiers......Page 413
17.2.1 Example: Classifying MNIST......Page 415
17.2.2 Example: Classifying CIFAR-10......Page 417
17.2.3 Quirks: Adversarial Examples......Page 423
17.3.5 Be Able to......Page 425
18.1 Image Classification......Page 427
18.1.1 Datasets for Classifying Images of Objects......Page 428
18.1.2 Datasets for Classifying Images of Scenes......Page 430
18.1.3 Augmentation and Ensembles......Page 431
18.1.4 AlexNet......Page 432
18.1.5 VGGNet......Page 434
18.1.6 Batch Normalization......Page 436
18.1.7 Computation Graphs......Page 437
18.1.8 Inception Networks......Page 438
18.1.9 Residual Networks......Page 440
18.2.1 How Object Detectors Work......Page 442
18.2.2 Selective Search......Page 444
18.2.3 R-CNN, Fast R-CNN and Faster R-CNN......Page 445
18.2.4 YOLO......Page 447
18.2.5 Evaluating Detectors......Page 449
18.3 Further Reading......Page 451
18.4.2 Remember These Facts......Page 453
18.4.3 Be Able to......Page 454
19.1 Better Low Dimensional Maps......Page 458
19.1.1 Sammon Mapping......Page 459
19.1.2 T-SNE......Page 460
19.2 Maps That Make Low-D Representations......Page 463
19.2.1 Encoders, Decoders, and Autoencoders......Page 464
19.2.2 Making Data Blocks Bigger......Page 465
19.2.3 The Denoising Autoencoder......Page 468
19.3 Generating Images from Examples......Page 472
19.3.1 Variational Autoencoders......Page 473
19.3.2 Adversarial Losses: Fooling a Classifier......Page 474
19.3.3 Matching Distributions with Test Functions......Page 476
19.3.4 Matching Distributions by Looking at Distances......Page 477
19.4.1 Remember These Terms......Page 478
19.4.3 Be Able to......Page 479
Index......Page 482
Procedures......Page 487
Worked Examples......Page 488
Remember this......Page 489