Statistical Foundations Of Data Science

Cover......Page 0
Half Title......Page 2
Title Page......Page 4
Copyright Page......Page 5
Dedication......Page 6
Table of Contents......Page 8
Preface......Page 18
1.1 Rise of Big Data and Dimensionality......Page 24
1.1.1 Biological sciences......Page 25
1.1.2 Health sciences......Page 27
1.1.3 Computer and information sciences......Page 28
1.1.4 Economics and finance......Page 30
1.2 Impact of Big Data......Page 32
1.3.1 Computation......Page 34
1.3.2 Noise accumulation......Page 35
1.3.3 Spurious correlation......Page 37
1.3.4 Statistical theory......Page 40
1.4 Aim of High-dimensional Statistical Learning......Page 41
1.6 Scope of the Book......Page 42
2.2 Multiple Linear Regression......Page 44
2.2.1 The Gauss-Markov theorem......Page 46
2.2.2 Statistical tests......Page 49
2.3 Weighted Least-Squares......Page 50
2.4 Box-Cox Transformation......Page 52
2.5 Model Building and Basis Expansions......Page 53
2.5.1 Polynomial regression......Page 54
2.5.2 Spline regression......Page 55
2.5.3 Multiple covariates......Page 58
2.6.1 Bias-variance tradeoff......Page 60
2.6.3 Bayesian interpretation......Page 61
2.6.4 Ridge regression solution path......Page 62
2.6.5 Kernel ridge regression......Page 64
2.7 Regression in Reproducing Kernel Hilbert Space......Page 65
2.8 Leave-one-out and Generalized Cross-validation......Page 70
2.9 Exercises......Page 72
3.1.1 Subset selection......Page 78
3.1.2 Relation with penalized regression......Page 79
3.1.3 Selection of regularization parameters......Page 80
3.2 Folded-concave Penalized Least Squares......Page 82
3.2.1 Orthonormal designs......Page 84
3.2.2 Penalty functions......Page 85
3.2.3 Thresholding by SCAD and MCP......Page 86
3.2.4 Risk properties......Page 87
3.2.5 Characterization of folded-concave PLS......Page 88
3.3.1 Nonnegative garrote......Page 89
3.3.2 Lasso......Page 91
3.3.3 Adaptive Lasso......Page 94
3.3.4 Elastic Net......Page 95
3.3.5 Dantzig selector......Page 97
3.3.6 SLOPE and sorted penalties......Page 100
3.3.7 Concentration inequalities and uniform convergence......Page 101
3.4.1 Bayesian view of the PLS......Page 104
3.4.2 A Bayesian framework for selection......Page 106
3.5.1 Quadratic programs......Page 107
3.5.2 Least angle regression∗......Page 109
3.5.3 Local quadratic approximations......Page 112
3.5.4 Local linear algorithm......Page 114
3.5.5 Penalized linear unbiased selection∗......Page 115
3.5.6 Cyclic coordinate descent algorithms......Page 116
3.5.7 Iterative shrinkage-thresholding algorithms......Page 117
3.5.9 ADMM......Page 119
3.5.10 Iterative local adaptive majorization and minimization......Page 120
3.5.11 Other methods and timeline......Page 121
3.6 Regularization Parameters for PLS......Page 122
3.6.1 Degrees of freedom......Page 123
3.6.3 Application to PLS estimators......Page 125
3.7.1 Residual variance of Lasso......Page 126
3.7.2 Refitted cross-validation......Page 127
3.8.1 Structured nonparametric models......Page 129
3.8.2 Group penalty......Page 130
3.9 Applications......Page 132
3.10 Bibliographical Notes......Page 137
3.11 Exercises......Page 138
4.1 Performance Benchmarks......Page 144
4.1.1 Performance measures......Page 145
4.1.2 Impact of model uncertainty......Page 148
4.1.2.1 Bayes lower bounds for orthogonal design......Page 149
4.1.2.2 Minimax lower bounds for general design......Page 153
4.1.3 Performance goals, sparsity and sub-Gaussian noise......Page 159
4.2 Penalized L0 Selection......Page 162
4.3 Lasso and Dantzig Selector......Page 168
4.3.1 Selection consistency......Page 169
4.3.2 Prediction and coefficient estimation errors......Page 173
4.3.3 Model size and least squares after selection......Page 184
4.3.4 Properties of the Dantzig selector......Page 190
4.3.5 Regularity conditions on the design matrix......Page 198
4.4 Properties of Concave PLS......Page 206
4.4.1 Properties of penalty functions......Page 208
4.4.2 Local and oracle solutions......Page 213
4.4.3 Properties of local solutions......Page 218
4.4.4 Global and approximate global solutions......Page 223
4.5 Smaller and Sorted Penalties......Page 229
4.5.1 Sorted concave penalties and their local approximation......Page 230
4.5.2 Approximate PLS with smaller and sorted penalties......Page 234
4.5.3 Properties of LLA and LCA......Page 243
4.6 Bibliographical Notes......Page 247
4.7 Exercises......Page 248
5.1.1 Exponential family......Page 250
5.1.2 Elements of generalized linear models......Page 253
5.1.3 Maximum likelihood......Page 254
5.1.4 Computing MLE: Iteratively reweighed least squares......Page 255
5.1.5 Deviance and analysis of deviance......Page 257
5.1.6 Residuals......Page 259
5.2.1 Bernoulli and binomial models......Page 261
5.2.2 Models for count responses......Page 264
5.3 Sparest Solution in High Confidence Set......Page 266
5.3.2 Examples......Page 267
5.3.3 Properties......Page 268
5.4 Variable Selection via Penalized Likelihood......Page 269
5.5.1 Local quadratic approximation......Page 272
5.5.2 Local linear approximation......Page 273
5.5.3 Coordinate descent......Page 274
5.6 Tuning Parameter Selection......Page 275
5.7 An Application......Page 277
5.8 Sampling Properties in Low-dimension......Page 279
5.8.1 Notation and regularity conditions......Page 280
5.8.2 The oracle property......Page 281
5.8.3 Sampling properties with diverging dimensions......Page 283
5.8.4 Asymptotic properties of GIC selectors......Page 285
5.9.1 The Lasso penalized estimator and its risk property......Page 287
5.9.2 Strong oracle property......Page 291
5.9.3 Numeric studies......Page 296
5.10 Risk Properties......Page 297
5.11 Bibliographical Notes......Page 301
5.12 Exercises......Page 303
6.1.1 Quantile regression......Page 310
6.1.2 Variable selection in quantile regression......Page 312
6.1.3 A fast algorithm for penalized quantile regression......Page 314
6.2 Penalized Composite Quantile Regression......Page 317
6.3.1 Robust regression......Page 320
6.3.2 Variable selection in Huber regression......Page 322
6.4 Rank Regression and Its Variable Selection......Page 324
6.4.2 Penalized weighted rank regression......Page 325
6.5 Variable Selection for Survival Data......Page 326
6.5.1 Partial likelihood......Page 328
6.5.2 Variable selection via penalized partial likelihood and its properties......Page 329
6.6 Theory of Folded-concave Penalized M-estimator......Page 331
6.6.1 Conditions on penalty and restricted strong convexity......Page 332
6.6.2 Statistical accuracy of penalized M-estimator with folded concave penalties......Page 333
6.6.3 Computational accuracy......Page 337
6.7 Bibliographical Notes......Page 340
6.8 Exercises......Page 342
7 High Dimensional Inference......Page 344
7.1 Inference in Linear Regression......Page 345
7.1.1 Debias of regularized regression estimators......Page 346
7.1.2 Choices of weights......Page 348
7.1.3 Inference for the noise level......Page 350
7.2 Inference in Generalized Linear Models......Page 353
7.2.1 Desparsified Lasso......Page 354
7.2.2 Decorrelated score estimator......Page 355
7.2.3 Test of linear hypotheses......Page 358
7.2.4 Numerical comparison......Page 360
7.2.5 An application......Page 361
7.3 Asymptotic Efficiency∗......Page 362
7.3.1 Statistical efficiency and Fisher information......Page 363
7.3.2 Linear regression with random design......Page 368
7.3.3 Partial linear regression......Page 374
7.4 Gaussian Graphical Models......Page 378
7.4.1 Inference via penalized least squares......Page 379
7.4.2 Sample size in regression and graphical models......Page 384
7.5.1 Local semi-LD decomposition......Page 391
7.5.2 Data swap......Page 393
7.5.3 Gradient approximation......Page 397
7.6 Bibliographical Notes......Page 399
7.7 Exercises......Page 400
8.1 Correlation Screening......Page 404
8.1.1 Sure screening property......Page 405
8.1.2 Connection to multiple comparison......Page 407
8.1.3 Iterative SIS......Page 408
8.2 Generalized and Rank Correlation Screening......Page 409
8.3.1 Generalized linear models......Page 412
8.3.2 A unified strategy for parametric feature screening......Page 414
8.3.3 Conditional sure independence screening......Page 417
8.4.1 Additive models......Page 418
8.4.2 Varying coefficient models......Page 419
8.4.3 Heterogeneous nonparametric models......Page 423
8.5.1 Sure independent ranking screening procedure......Page 424
8.5.2 Feature screening via distance correlation......Page 426
8.5.3 Feature screening for high-dimensional categorial data......Page 429
8.6.1 Feature screening via forward regression......Page 432
8.6.2 Sparse maximum likelihood estimate......Page 433
8.6.3 Feature screening via partial correlation......Page 435
8.7.1 RCV algorithm......Page 440
8.7.2 RCV in linear models......Page 441
8.7.3 RCV in nonparametric regression......Page 443
8.8 An Illustration......Page 446
8.9 Bibliographical Notes......Page 449
8.10 Exercises......Page 451
9.1 Basic Facts about Matrices......Page 454
9.2.1 Covariance regularization by thresholding and banding......Page 458
9.2.2 Asymptotic properties......Page 461
9.2.3 Nearest positive definite matrices......Page 464
9.3 Robust Covariance Inputs......Page 466
9.4.1 Gaussian graphical models......Page 469
9.4.2 Penalized likelihood and M-estimation......Page 470
9.4.3 Penalized least-squares......Page 471
9.4.4 CLIME and its adaptive version......Page 474
9.5 Latent Gaussian Graphical Models......Page 479
9.6.1 Proof of Theorem 9.1......Page 483
9.6.2 Proof of Theorem 9.3......Page 484
9.6.3 Proof of Theorem 9.4......Page 485
9.6.4 Proof of Theorem 9.6......Page 486
9.7 Bibliographical Notes......Page 488
9.8 Exercises......Page 489
10.1.1 Introduction to PCA......Page 494
10.1.2 Power method......Page 496
10.2 Factor Models and Structured Covariance Learning......Page 497
10.2.1 Factor model and high-dimensional PCA......Page 498
10.2.2 Extracting latent factors and POET......Page 501
10.2.3 Methods for selecting number of factors......Page 503
10.3.1 Factor model with observable factors......Page 506
10.3.2 Robust initial estimation of covariance matrix......Page 508
10.4 Augmented Factor Models and Projected PCA......Page 511
10.5.1 Properties for estimating loading matrix......Page 514
10.5.2 Properties for estimating covariance matrices......Page 516
10.5.3 Properties for estimating realized latent factors......Page 517
10.6.1 Proof of Theorem 10.1......Page 518
10.6.2 Proof of Theorem 10.2......Page 523
10.6.3 Proof of Theorem 10.3......Page 524
10.6.4 Proof of Theorem 10.4......Page 527
10.7 Bibliographical Notes......Page 529
10.8 Exercises......Page 530
11.1 Factor-adjusted Regularized Model Selection......Page 534
11.1.1 Importance of factor adjustments......Page 535
11.1.2 FarmSelect......Page 536
11.1.3 Application to forecasting bond risk premia......Page 537
11.1.4 Application to a neuroblastoma data......Page 539
11.2 Factor-adjusted Robust Multiple Testing......Page 541
11.2.1 False discovery rate control......Page 542
11.2.2 Multiple testing under dependence measurements......Page 544
11.2.3 Power of factor adjustments......Page 546
11.2.4 FarmTest......Page 547
11.2.5 Application to neuroblastoma data......Page 549
11.3.1 Principal component regression......Page 551
11.3.2 Augmented principal component regression......Page 553
11.3.3 Application to forecast bond risk premia......Page 554
11.4 Applications to Statistical Machine Learning......Page 555
11.4.1 Community detection......Page 556
11.4.2 Topic model......Page 562
11.4.3 Matrix completion......Page 563
11.4.4 Item ranking......Page 565
11.4.5 Gaussian mixture models......Page 568
11.5 Bibliographical Notes......Page 571
11.6 Exercises......Page 573
12.1.1 Linear and quadratic discriminant analysis......Page 576
12.1.2 Logistic regression......Page 580
12.2 Kernel Density Classifiers and Naive Bayes......Page 582
12.3 Nearest Neighbor Classifiers......Page 586
12.4.1 Classification trees......Page 588
12.4.2 Bagging......Page 590
12.4.3 Random forests......Page 592
12.4.4 Boosting......Page 594
12.5.1 The standard support vector machine......Page 598
12.5.2 Generalizations of SVMs......Page 601
12.6.1 The importance of sparsity under high-dimensionality......Page 604
12.6.2 Sparse support vector machines......Page 606
12.6.3 Sparse large margin classifiers......Page 607
12.7 Sparse Discriminant Analysis......Page 609
12.7.1 Nearest shrunken centroids classifier......Page 611
12.7.2 Features annealed independent rule......Page 612
12.7.3 Selection bias of sparse independence rules......Page 614
12.7.4 Regularized optimal affine discriminant......Page 615
12.7.5 Linear programming discriminant......Page 616
12.7.6 Direct sparse discriminant analysis......Page 617
12.7.7 Solution path equivalence between ROAD and DSDA......Page 619
12.8.1 Feature augmentation......Page 620
12.8.2 Penalized additive logistic regression......Page 622
12.8.3 Semiparametric sparse discriminant analysis......Page 623
12.10 Exercises......Page 625
13.1 Cluster Analysis......Page 630
13.1.1 K-means clustering......Page 631
13.1.2 Hierarchical clustering......Page 632
13.1.3 Model-based clustering......Page 634
13.1.4 Spectral clustering......Page 638
13.2 Data-driven Choices of the Number of Clusters......Page 640
13.3.1 Sparse clustering......Page 643
13.3.2 Sparse model-based clustering......Page 645
13.3.3 Sparse mixture of experts model......Page 647
13.4.1 Inconsistency of the regular PCA......Page 650
13.4.2 Consistency under sparse eigenvector model......Page 651
13.5.1 Sparse PCA......Page 653
13.5.2 An iterative SVD thresholding approach......Page 656
13.5.3 A penalized matrix decomposition approach......Page 658
13.5.4 A semidefinite programming approach......Page 659
13.5.5 A generalized power method......Page 660
13.6 Bibliographical Notes......Page 662
13.7 Exercises......Page 663
14 An Introduction to Deep Learning......Page 666
14.1 Rise of Deep Learning......Page 667
14.2.1 Model setup......Page 669
14.2.2 Back-propagation in computational graphs......Page 670
14.3 Popular Models......Page 673
14.3.1 Convolutional neural networks......Page 674
14.3.2.1 Vanilla RNNs......Page 677
14.3.2.2 GRUs and LSTM......Page 678
14.3.2.3 Multilayer RNNs......Page 679
14.3.3 Modules......Page 680
14.4.1 Autoencoders......Page 682
14.4.2.1 Sampling view of GANs......Page 685
14.4.2.2 Minimum distance view of GANs......Page 686
14.5 Training deep neural nets......Page 688
14.5.1.1 Mini-batch SGD......Page 689
14.5.1.3 SGD with adaptive learning rates......Page 690
14.5.2.1 ReLU activation function......Page 691
14.5.2.3 Batch normalization......Page 692
14.5.3.2 Dropout......Page 693
14.6 Example: Image Classification......Page 694
14.7 Additional Examples using FensorFlow and R......Page 696
14.8 Bibliography Notes......Page 703
References......Page 706
Author Index......Page 754
Index......Page 766