Data Science and Machine Learning: Mathematical and Statistical Methods

Cover......Page 1
Half Title......Page 2
Title Page......Page 4
Copyright Page......Page 5
Dedication......Page 6
Table of Contents......Page 8
Preface......Page 14
Notation......Page 18
 1.1 Introduction......Page 24
 1.2 Structuring Features According to Type......Page 26
 1.3 Summary Tables......Page 29
 1.4 Summary Statistics......Page 30
 1.5 Visualizing Data......Page 31
  1.5.2 Plotting Quantitative Variables......Page 32
  1.5.3 Data Visualization in a Bivariate Setting......Page 35
 Exercises.......Page 38
 2.1 Introduction......Page 42
 2.2 Supervised and Unsupervised Learning......Page 43
 2.3 Training and Test Loss......Page 46
 2.4 Tradeoffs in Statistical Learning......Page 54
  2.5.1 In-Sample Risk......Page 58
  2.5.2 Cross-Validation......Page 60
 2.6 Modeling Data......Page 63
 2.7 Multivariate Normal Models......Page 67
 2.8 Normal Linear Models......Page 69
 2.9 Bayesian Learning......Page 70
 Exercises......Page 81
 3.1 Introduction......Page 90
  3.2.1 Generating Random Numbers......Page 91
  3.2.2 Simulating Random Variables......Page 92
  3.2.3 Simulating Random Vectors and Processes......Page 97
  3.2.4 Resampling......Page 99
  3.2.5 Markov Chain Monte Carlo......Page 101
  3.3.1 Crude Monte Carlo......Page 108
  3.3.2 Bootstrap Method......Page 111
  3.3.3 Variance Reduction......Page 115
  3.4.1 Simulated Annealing......Page 119
  3.4.2 Cross-Entropy Method......Page 123
  3.4.3 Splitting for Optimization......Page 126
  3.4.4 Noisy Optimization......Page 128
 Exercises......Page 136
 4.1 Introduction......Page 144
 4.2 Risk and Loss in Unsupervised Learning......Page 145
 4.3 Expectation–Maximization (EM) Algorithm......Page 151
 4.4 Empirical Distribution and Density Estimation......Page 154
  4.5.1 Mixture Models......Page 158
  4.5.2 EM Algorithm for Mixture Models......Page 160
 4.6 Clustering via Vector Quantization......Page 165
  4.6.1 K-Means......Page 167
  4.6.2 Clustering via Continuous Multiextremal Optimization......Page 169
 4.7 Hierarchical Clustering......Page 170
  4.8.1 Motivation: Principal Axes of an Ellipsoid......Page 176
  4.8.2 PCA and Singular Value Decomposition (SVD)......Page 178
 Exercises......Page 183
 5.1 Introduction......Page 190
 5.2 Linear Regression......Page 192
  5.3.1 Parameter Estimation......Page 194
  5.3.2 Model Selection and Prediction......Page 195
  5.3.3 Cross-Validation and Predictive Residual Sum of Squares......Page 196
  5.3.4 In-Sample Risk and Akaike Information Criterion......Page 198
  5.3.5 Categorical Features......Page 200
  5.3.6 Nested Models......Page 203
  5.3.7 Coefficient of Determination......Page 204
 5.4 Inference for Normal Linear Models......Page 205
  5.4.1 Comparing Two Normal Linear Models......Page 206
  5.4.2 Confidence and Prediction Intervals......Page 209
 5.5 Nonlinear Regression Models......Page 211
  5.6.1 Modeling......Page 214
  5.6.2 Analysis......Page 216
  5.6.3 Analysis of Variance (ANOVA)......Page 218
  5.6.5 Model Validation......Page 221
  5.6.6 Variable Selection......Page 222
 5.7 Generalized Linear Models......Page 227
 Exercises......Page 230
 6.1 Introduction......Page 238
 6.2 Regularization......Page 239
 6.3 Reproducing Kernel Hilbert Spaces......Page 245
  6.4.1 Reproducing Kernels via Feature Mapping......Page 247
  6.4.2 Kernels from Characteristic Functions......Page 248
  6.4.3 Reproducing Kernels Using Orthonormal Features......Page 250
  6.4.4 Kernels from Kernels......Page 252
 6.5 Representer Theorem......Page 253
 6.6 Smoothing Cubic Splines......Page 258
 6.7 Gaussian Process Regression......Page 261
 6.8 Kernel PCA......Page 265
 Exercises......Page 268
 7.1 Introduction......Page 274
 7.2 Classification Metrics......Page 276
 7.3 Classification via Bayes’ Rule......Page 280
 7.4 Linear and Quadratic Discriminant Analysis......Page 282
 7.5 Logistic Regression and Softmax Classification......Page 289
 7.6 K-Nearest Neighbors Classification......Page 291
 7.7 Support Vector Machine......Page 292
 7.8 Classification with Scikit-Learn......Page 300
 Exercises......Page 302
 8.1 Introduction......Page 310
 8.2 Top-Down Construction of Decision Trees......Page 312
  8.2.1 Regional Prediction Functions......Page 313
  8.2.2 Splitting Rules......Page 314
  8.2.3 Termination Criterion......Page 315
  8.2.4 Basic Implementation......Page 317
  8.3.3 Alternative Splitting Rules......Page 321
  8.3.5 Missing Values......Page 322
 8.4 Controlling the Tree Shape......Page 323
  8.4.1 Cost-Complexity Pruning......Page 326
  8.4.2 Advantages and Limitations of Decision Trees......Page 327
 8.5 Bootstrap Aggregation......Page 328
 8.6 Random Forests......Page 332
 8.7 Boosting......Page 336
 Exercises......Page 344
 9.1 Introduction......Page 346
 9.2 Feed-Forward Neural Networks......Page 349
 9.3 Back-Propagation......Page 353
  9.4.1 Steepest Descent......Page 357
  9.4.2 Levenberg–Marquardt Method......Page 358
  9.4.3 Limited-Memory BFGS Method......Page 359
  9.4.4 Adaptive Gradient Methods......Page 361
  9.5.1 Simple Polynomial Regression......Page 363
  9.5.2 Image Classification......Page 367
 Exercises......Page 372
 A.1 Vector Spaces, Bases, and Matrices......Page 378
 A.2 Inner Product......Page 383
 A.3 Complex Vectors and Matrices......Page 384
 A.4 Orthogonal Projections......Page 385
 A.5 Eigenvalues and Eigenvectors......Page 386
  A.5.1 Left- and Right-Eigenvectors......Page 387
  A.6.1 (P)LU Decomposition......Page 391
  A.6.2 Woodbury Identity......Page 393
  A.6.3 Cholesky Decomposition......Page 396
  A.6.4 QR Decomposition and the Gram–Schmidt Procedure......Page 398
  A.6.5 Singular Value Decomposition......Page 399
  A.6.6 Solving Structured Matrix Equations......Page 402
 A.7 Functional Analysis......Page 407
 A.8 Fourier Transforms......Page 413
  A.8.1 Discrete Fourier Transform......Page 415
  A.8.2 Fast Fourier Transform......Page 417
 B.1 Multivariate Differentiation......Page 420
  B.1.2 Chain Rule......Page 423
 B.2 Optimization Theory......Page 425
  B.2.1 Convexity and Optimization......Page 426
  B.2.2 Lagrangian Method......Page 429
  B.2.3 Duality......Page 430
 B.3 Numerical Root-Finding and Minimization......Page 431
  B.3.1 Newton-Like Methods......Page 432
  B.3.2 Quasi-Newton Methods......Page 434
  B.3.3 Normal Approximation Method......Page 436
  B.3.4 Nonlinear Least Squares......Page 437
 B.4 Constrained Minimization via Penalty Functions......Page 438
 C.1 Random Experiments and Probability Spaces......Page 444
 C.2 Random Variables and Probability Distributions......Page 445
 C.3 Expectation......Page 449
 C.4 Joint Distributions......Page 450
  C.5.2 Independence......Page 451
  C.5.3 Expectation and Covariance......Page 452
  C.5.4 Conditional Density and Conditional Expectation ......Page 453
 C.6 Functions of Random Variables......Page 454
 C.7 Multivariate Normal Distribution......Page 457
 C.8 Convergence of Random Variables......Page 462
 C.9 Law of Large Numbers and Central Limit Theorem......Page 468
 C.10 Markov Chains......Page 474
 C.11 Statistics......Page 476
 C.12 Estimation......Page 477
  C.12.1 Method of Moments......Page 478
  C.12.2 Maximum Likelihood Method......Page 479
 C.13 Confidence Intervals......Page 480
 C.14 Hypothesis Testing......Page 481
 D.1 Getting Started......Page 486
 D.2 Python Objects......Page 488
 D.3 Types and Operators......Page 489
 D.4 Functions and Methods......Page 491
 D.5 Modules......Page 492
 D.6 Flow Control......Page 494
 D.7 Iteration......Page 495
 D.8 Classes......Page 496
 D.9 Files......Page 498
  D.10.1 Creating and Shaping Arrays......Page 501
  D.10.3 Array Operations......Page 503
  D.10.4 Random Numbers......Page 505
  D.11.1 Creating a Basic Plot......Page 506
  D.12.1 Series and DataFrame......Page 508
  D.12.2 Manipulating Data Frames......Page 510
  D.12.3 Extracting Information......Page 511
 D.13 Scikit-learn......Page 513
  D.13.2 Standardization......Page 514
  D.13.4 Testing the Model......Page 515
 D.14 System Calls, URL Access, and Speed-Up......Page 516
Bibliography......Page 518
Index......Page 526