Machine Learning Refined: Foundations, Algorithms, and Applications

Contents......Page 6
Preface......Page 11
Acknowledgements......Page 21
 1.2 Distinguishing Cats from Dogs: a Machine Learning Approach......Page 22
 1.3 The Basic Taxonomy of Machine Learning Problems......Page 27
 1.4 Mathematical Optimization......Page 37
 1.5 Conclusion......Page 39
Part I Mathematical Optimization......Page 40
  2.1 Introduction......Page 42
  2.2 The Zero-Order Optimality Condition......Page 44
  2.3 Global Optimization Methods......Page 45
  2.4 Local Optimization Methods......Page 48
  2.5 Random Search......Page 52
  2.6 Coordinate Search and Descent......Page 60
  2.7 Conclusion......Page 61
  2.8 Exercises......Page 63
  3.2 The First-Order Optimality Condition......Page 66
  3.3 The Geometry of First-Order Taylor Series......Page 73
  3.4 Computing Gradients Efficiently......Page 76
  3.5 Gradient Descent......Page 77
  3.6 Two Natural Weaknesses of Gradient Descent......Page 86
  3.8 Exercises......Page 92
  4.1 The Second-Order Optimality Condition......Page 96
  4.2 The Geometry of Second-Order Taylor Series......Page 99
  4.3 Newton’s Method......Page 102
  4.4 Two Natural Weaknesses of Newton’s Method......Page 111
  4.5 Conclusion......Page 112
  4.6 Exercises......Page 113
Part II Linear Learning......Page 118
  5.2 Least Squares Linear Regression......Page 119
  5.3 Least Absolute Deviations......Page 128
  5.4 Regression Quality Metrics......Page 131
  5.5 Weighted Regression......Page 133
  5.6 Multi-Output Regression......Page 136
  5.7 Conclusion......Page 140
  5.8 Exercises......Page 141
  5.9 Endnotes......Page 144
  6.2 Logistic Regression and the Cross Entropy Cost......Page 145
  6.3 Logistic Regression and the Softmax Cost......Page 155
  6.4 The Perceptron......Page 160
  6.5 Support Vector Machines......Page 170
  6.6 Which Approach Produces the Best Results?......Page 177
  6.7 The Categorical Cross Entropy Cost......Page 178
  6.8 Classification Quality Metrics......Page 180
  6.9 Weighted Two-Class Classification......Page 187
  6.10 Conclusion......Page 190
  6.11 Exercises......Page 191
  7.2 One-versus-All Multi-Class Classification......Page 194
  7.3 Multi-Class Classification and the Perceptron......Page 204
  7.4 Which Approach Produces the Best Results?......Page 212
  7.5 The Categorical Cross Entropy Cost Function......Page 213
  7.6 Classification Quality Metrics......Page 218
  7.7 Weighted Multi-Class Classification......Page 222
  7.8 Stochastic and Mini-Batch Learning......Page 223
  7.10 Exercises......Page 225
  8.2 Fixed Spanning Sets, Orthonormality, and Projections......Page 228
  8.3 The Linear Autoencoder and Principal Component Analysis......Page 233
  8.4 Recommender Systems......Page 239
  8.5 K-Means Clustering......Page 241
  8.6 General Matrix Factorization Techniques......Page 247
  8.7 Conclusion......Page 250
  8.8 Exercises......Page 251
  8.9 Endnotes......Page 253
  9.1 Introduction......Page 257
  9.2 Histogram Features......Page 258
  9.3 Feature Scaling via Standard Normalization......Page 269
  9.4 Imputing Missing Values in a Dataset......Page 274
  9.5 Feature Scaling via PCA-Sphering......Page 275
  9.6 Feature Selection via Boosting......Page 278
  9.7 Feature Selection via Regularization......Page 284
  9.9 Exercises......Page 289
Part III Nonlinear Learning......Page 293
  10.2 Nonlinear Regression......Page 295
  10.3 Nonlinear Multi-Output Regression......Page 302
  10.4 Nonlinear Two-Class Classification......Page 306
  10.5 Nonlinear Multi-Class Classification......Page 310
  10.6 Nonlinear Unsupervised Learning......Page 314
  10.8 Exercises......Page 318
  11.1 Introduction......Page 324
  11.2 Universal Approximators......Page 327
  11.3 Universal Approximation of Real Data......Page 343
  11.4 Naive Cross-Validation......Page 355
  11.5 Efficient Cross-Validation via Boosting......Page 360
  11.6 Efficient Cross-Validation via Regularization......Page 370
  11.7 Testing Data......Page 381
  11.8 Which Universal Approximator Works Best in Practice?......Page 385
  11.9 Bagging Cross-Validated Models......Page 386
  11.10 K-Fold Cross-Validation......Page 393
  11.11 When Feature Learning Fails......Page 398
  11.12 Conclusion......Page 399
  11.13 Exercises......Page 400
  12.2 Fixed-Shape Universal Approximators......Page 403
  12.3 The Kernel Trick......Page 406
  12.4 Kernels as Measures of Similarity......Page 416
  12.5 Optimization of Kernelized Models......Page 417
  12.6 Cross-Validating Kernelized Learners......Page 418
  12.8 Exercises......Page 419
  13.2 Fully Connected Neural Networks......Page 423
  13.3 Activation Functions......Page 444
  13.4 The Backpropagation Algorithm......Page 447
  13.5 Optimization of Neural Network Models......Page 448
  13.6 Batch Normalization......Page 450
  13.7 Cross-Validation via Early Stopping......Page 458
  13.8 Conclusion......Page 460
  13.9 Exercises......Page 461
  14.2 From Stumps to Deep Trees......Page 463
  14.3 Regression Trees......Page 466
  14.4 Classification Trees......Page 472
  14.5 Gradient Boosting......Page 478
  14.6 Random Forests......Page 482
  14.7 Cross-Validation Techniques for Recursively Defined Trees......Page 484
  14.9 Exercises......Page 487
Part IV Appendices......Page 491
  A.2 Momentum-Accelerated Gradient Descent......Page 493
  A.3 Normalized Gradient Descent......Page 498
  A.4 Advanced Gradient-Based Methods......Page 505
  A.5 Mini-Batch Optimization......Page 507
  A.6 Conservative Steplength Rules......Page 510
  A.7 Newton’s Method, Regularization, and Nonconvex Functions......Page 519
  A.8 Hessian-Free Methods......Page 522
  B.2 The Derivative......Page 531
  B.3 Derivative Rules for Elementary Functions and Operations......Page 534
  B.4 The Gradient......Page 536
  B.5 The Computation Graph......Page 537
  B.6 The Forward Mode of Automatic Differentiation......Page 540
  B.7 The Reverse Mode of Automatic Differentiation......Page 546
  B.8 Higher-Order Derivatives......Page 549
  B.9 Taylor Series......Page 551
  B.10 Using the autograd Library......Page 556
  C.2 Vectors and Vector Operations......Page 566
  C.3 Matrices and Matrix Operations......Page 573
  C.4 Eigenvalues and Eigenvectors......Page 576
  C.5 Vector and Matrix Norms......Page 579
References......Page 584
Index......Page 589