Mathematics for machine learning

Foreword......Page 7
Part I Mathematical Foundations......Page 15
 1 Introduction and Motivation......Page 17
  1.1 Finding Words for Intuitions......Page 18
  1.2 Two Ways to Read This Book......Page 19
  1.3 Exercises and Feedback......Page 22
 2 Linear Algebra......Page 23
  2.1 Systems of Linear Equations......Page 25
  2.2 Matrices......Page 28
  2.3 Solving Systems of Linear Equations......Page 33
  2.4 Vector Spaces......Page 41
  2.5 Linear Independence......Page 46
  2.6 Basis and Rank......Page 50
  2.7 Linear Mappings......Page 54
  2.8 Affine Spaces......Page 67
 Exercises......Page 69
 3 Analytic Geometry......Page 76
  3.1 Norms......Page 77
  3.2 Inner Products......Page 78
  3.3 Lengths and Distances......Page 81
  3.4 Angles and Orthogonality......Page 82
  3.5 Orthonormal Basis......Page 84
  3.6 Orthogonal Complement......Page 85
  3.7 Inner Product of Functions......Page 86
  3.8 Orthogonal Projections......Page 87
  3.9 Rotations......Page 97
  3.10 Further Reading......Page 100
 Exercises......Page 101
 4 Matrix Decompositions......Page 104
  4.1 Determinant and Trace......Page 105
  4.2 Eigenvalues and Eigenvectors......Page 111
  4.3 Cholesky Decomposition......Page 120
  4.4 Eigendecomposition and Diagonalization......Page 121
  4.5 Singular Value Decomposition......Page 125
  4.6 Matrix Approximation......Page 135
  4.7 Matrix Phylogeny......Page 140
  4.8 Further Reading......Page 141
 Exercises......Page 143
 5 Vector Calculus......Page 145
  5.1 Differentiation of Univariate Functions......Page 147
  5.2 Partial Differentiation and Gradients......Page 152
  5.3 Gradients of Vector-Valued Functions......Page 155
  5.4 Gradients of Matrices......Page 161
  5.5 Useful Identities for Computing Gradients......Page 164
  5.6 Backpropagation and Automatic Differentiation......Page 165
  5.7 Higher-Order Derivatives......Page 170
  5.8 Linearization and Multivariate Taylor Series......Page 171
 Exercises......Page 176
  6.1 Construction of a Probability Space......Page 178
  6.2 Discrete and Continuous Probabilities......Page 184
  6.3 Sum Rule, Product Rule, and Bayes' Theorem......Page 189
  6.4 Summary Statistics and Independence......Page 192
  6.5 Gaussian Distribution......Page 203
  6.6 Conjugacy and the Exponential Family......Page 211
  6.7 Change of Variables/Inverse Transform......Page 220
  6.8 Further Reading......Page 227
 Exercises......Page 228
 7 Continuous Optimization......Page 231
  7.1 Optimization Using Gradient Descent......Page 233
  7.2 Constrained Optimization and Lagrange Multipliers......Page 239
  7.3 Convex Optimization......Page 242
  7.4 Further Reading......Page 252
 Exercises......Page 253
Part II Central Machine Learning Problems......Page 255
  8.1 Data, Models, and Learning......Page 257
  8.2 Empirical Risk Minimization......Page 264
  8.3 Parameter Estimation......Page 271
  8.4 Probabilistic Modeling and Inference......Page 278
  8.5 Directed Graphical Models......Page 284
  8.6 Model Selection......Page 289
 9 Linear Regression......Page 295
  9.1 Problem Formulation......Page 297
  9.2 Parameter Estimation......Page 298
  9.3 Bayesian Linear Regression......Page 309
  9.4 Maximum Likelihood as Orthogonal Projection......Page 319
  9.5 Further Reading......Page 321
 10 Dimensionality Reduction with Principal Component Analysis......Page 323
  10.1 Problem Setting......Page 324
  10.2 Maximum Variance Perspective......Page 326
  10.3 Projection Perspective......Page 331
  10.4 Eigenvector Computation and Low-Rank Approximations......Page 339
  10.5 PCA in High Dimensions......Page 341
  10.6 Key Steps of PCA in Practice......Page 342
  10.7 Latent Variable Perspective......Page 345
  10.8 Further Reading......Page 349
 11 Density Estimation with Gaussian Mixture Models......Page 354
  11.1 Gaussian Mixture Model......Page 355
  11.2 Parameter Learning via Maximum Likelihood......Page 356
  11.3 EM Algorithm......Page 366
  11.4 Latent-Variable Perspective......Page 369
  11.5 Further Reading......Page 374
 12 Classification with Support Vector Machines......Page 376
  12.1 Separating Hyperplanes......Page 378
  12.2 Primal Support Vector Machine......Page 380
  12.3 Dual Support Vector Machine......Page 389
  12.4 Kernels......Page 394
  12.5 Numerical Solution......Page 396
  12.6 Further Reading......Page 398
 References......Page 401
 Index......Page 413