An Introduction to Statistical Learning (with Applications in R)

Preface......Page 8
Contents......Page 10
1 Introduction......Page 16
2.1 What Is Statistical Learning?......Page 30
2.1.1 Why Estimate f?......Page 32
2.1.2 How Do We Estimate f?......Page 36
2.1.3 The Trade-Off Between Prediction Accuracyand Model Interpretability......Page 39
2.1.4 Supervised Versus Unsupervised Learning......Page 41
2.1.5 Regression Versus Classification Problems......Page 43
2.2.1 Measuring the Quality of Fit......Page 44
2.2.2 The Bias-Variance Trade-Off......Page 48
2.2.3 The Classification Setting......Page 52
2.3.1 Basic Commands......Page 57
2.3.2 Graphics......Page 60
2.3.3 Indexing Data......Page 62
2.3.4 Loading Data......Page 63
2.3.5 Additional Graphical and Numerical Summaries......Page 64
2.4 Exercises......Page 67
3 Linear Regression......Page 73
3.1 Simple Linear Regression......Page 74
3.1.1 Estimating the Coefficients......Page 75
3.1.2 Assessing the Accuracy of the CoefficientEstimates......Page 77
Residual Standard Error......Page 82
R2 Statistic......Page 83
3.2 Multiple Linear Regression......Page 85
3.2.1 Estimating the Regression Coefficients......Page 86
3.2.2 Some Important Questions......Page 88
One: Is There a Relationship Between the Response and Predictors?......Page 89
Two: Deciding on Important Variables......Page 92
Three: Model Fit......Page 93
Four: Predictions......Page 95
3.3.1 Qualitative Predictors......Page 96
Predictors with Only Two Levels......Page 98
Qualitative Predictors with More than Two Levels......Page 99
3.3.2 Extensions of the Linear Model......Page 100
Removing the Additive Assumption......Page 101
Non-linear Relationships......Page 104
1. Non-linearity of the Data......Page 106
2. Correlation of Error Terms......Page 107
3. Non-constant Variance of Error Terms......Page 109
4. Outliers......Page 110
5. High Leverage Points......Page 111
6. Collinearity......Page 113
3.4 The Marketing Plan......Page 116
3.5 Comparison of Linear Regression with K-NearestNeighbors......Page 118
3.6.1 Libraries......Page 123
3.6.2 Simple Linear Regression......Page 124
3.6.3 Multiple Linear Regression......Page 127
3.6.5 Non-linear Transformations of the Predictors......Page 129
3.6.6 Qualitative Predictors......Page 131
3.6.7 Writing Functions......Page 133
3.7 Exercises......Page 134
4.1 An Overview of Classification......Page 141
4.2 Why Not Linear Regression?......Page 142
4.3 Logistic Regression......Page 144
4.3.1 The Logistic Model......Page 145
4.3.2 Estimating the Regression Coefficients......Page 147
4.3.3 Making Predictions......Page 148
4.3.4 Multiple Logistic Regression......Page 149
4.3.5 Logistic Regression for >2 Response Classes......Page 151
4.4.1 Using Bayes\' Theorem for Classification......Page 152
4.4.2 Linear Discriminant Analysis for p=1......Page 153
4.4.3 Linear Discriminant Analysis for p>1......Page 156
4.4.4 Quadratic Discriminant Analysis......Page 163
4.5 A Comparison of Classification Methods......Page 164
4.6.1 The Stock Market Data......Page 168
4.6.2 Logistic Regression......Page 170
4.6.3 Linear Discriminant Analysis......Page 175
4.6.5 K-Nearest Neighbors......Page 177
4.6.6 An Application to Caravan Insurance Data......Page 179
4.7 Exercises......Page 181
5 Resampling Methods......Page 188
5.1.1 The Validation Set Approach......Page 189
5.1.2 Leave-One-Out Cross-Validation......Page 191
5.1.3 k-Fold Cross-Validation......Page 194
5.1.4 Bias-Variance Trade-Off for k-FoldCross-Validation......Page 196
5.1.5 Cross-Validation on Classification Problems......Page 197
5.2 The Bootstrap......Page 199
5.3.1 The Validation Set Approach......Page 203
5.3.2 Leave-One-Out Cross-Validation......Page 205
5.3.3 k-Fold Cross-Validation......Page 206
Estimating the Accuracy of a Statistic of Interest......Page 207
Estimating the Accuracy of a Linear Regression Model......Page 208
5.4 Exercises......Page 210
6 Linear Model Selection and Regularization......Page 215
6.1.1 Best Subset Selection......Page 217
Forward Stepwise Selection......Page 219
Backward Stepwise Selection......Page 220
Cp, AIC, BIC, and Adjusted R2......Page 222
Validation and Cross-Validation......Page 225
6.2 Shrinkage Methods......Page 226
6.2.1 Ridge Regression......Page 227
An Application to the Credit Data......Page 228
Why Does Ridge Regression Improve Over Least Squares?......Page 229
6.2.2 The Lasso......Page 231
Another Formulation for Ridge Regression and the Lasso......Page 232
The Variable Selection Property of the Lasso......Page 233
Comparing the Lasso and Ridge Regression......Page 235
A Simple Special Case for Ridge Regression and the Lasso......Page 236
Bayesian Interpretation for Ridge Regression and the Lasso......Page 238
6.2.3 Selecting the Tuning Parameter......Page 239
6.3 Dimension Reduction Methods......Page 240
6.3.1 Principal Components Regression......Page 242
An Overview of Principal Components Analysis......Page 243
The Principal Components Regression Approach......Page 245
6.3.2 Partial Least Squares......Page 249
6.4.1 High-Dimensional Data......Page 250
6.4.2 What Goes Wrong in High Dimensions?......Page 251
6.4.3 Regression in High Dimensions......Page 253
6.4.4 Interpreting Results in High Dimensions......Page 255
6.5.1 Best Subset Selection......Page 256
6.5.2 Forward and Backward Stepwise Selection......Page 259
6.5.3 Choosing Among Models Using the ValidationSet Approach and Cross-Validation......Page 260
6.6.1 Ridge Regression......Page 263
6.6.2 The Lasso......Page 267
6.7.1 Principal Components Regression......Page 268
6.7.2 Partial Least Squares......Page 270
6.8 Exercises......Page 271
7 Moving Beyond Linearity......Page 277
7.1 Polynomial Regression......Page 278
7.2 Step Functions......Page 280
7.4 Regression Splines......Page 282
7.4.2 Constraints and Splines......Page 283
7.4.3 The Spline Basis Representation......Page 285
7.4.4 Choosing the Number and Locationsof the Knots......Page 286
7.4.5 Comparison to Polynomial Regression......Page 288
7.5.1 An Overview of Smoothing Splines......Page 289
7.5.2 Choosing the Smoothing Parameter......Page 290
7.6 Local Regression......Page 292
7.7 Generalized Additive Models......Page 294
7.7.1 GAMs for Regression Problems......Page 295
Pros and Cons of GAMs......Page 297
7.7.2 GAMs for Classification Problems......Page 298
7.8.1 Polynomial Regression and Step Functions......Page 299
7.8.2 Splines......Page 304
7.8.3 GAMs......Page 306
7.9 Exercises......Page 309
8.1.1 Regression Trees......Page 314
Predicting Baseball Players\' Salaries Using Regression Trees......Page 315
Prediction via Stratification of the Feature Space......Page 317
Tree Pruning......Page 318
8.1.2 Classification Trees......Page 322
8.1.3 Trees Versus Linear Models......Page 325
8.1.4 Advantages and Disadvantages of Trees......Page 326
8.2.1 Bagging......Page 327
Out-of-Bag Error Estimation......Page 328
Variable Importance Measures......Page 329
8.2.2 Random Forests......Page 330
8.2.3 Boosting......Page 332
8.3.1 Fitting Classification Trees......Page 335
8.3.2 Fitting Regression Trees......Page 338
8.3.3 Bagging and Random Forests......Page 339
8.3.4 Boosting......Page 341
8.4 Exercises......Page 343
9.1 Maximal Margin Classifier......Page 347
9.1.1 What Is a Hyperplane?......Page 348
9.1.2 Classification Using a Separating Hyperplane......Page 349
9.1.3 The Maximal Margin Classifier......Page 351
9.1.4 Construction of the Maximal Margin Classifier......Page 352
9.1.5 The Non-separable Case......Page 353
9.2.1 Overview of the Support Vector Classifier......Page 354
9.2.2 Details of the Support Vector Classifier......Page 355
9.3.1 Classification with Non-linear DecisionBoundaries......Page 359
9.3.2 The Support Vector Machine......Page 360
9.3.3 An Application to the Heart Disease Data......Page 363
9.4.1 One-Versus-One Classification......Page 365
9.5 Relationship to Logistic Regression......Page 366
9.6.1 Support Vector Classifier......Page 369
9.6.2 Support Vector Machine......Page 373
9.6.3 ROC Curves......Page 375
9.6.5 Application to Gene Expression Data......Page 376
9.7 Exercises......Page 378
10.1 The Challenge of Unsupervised Learning......Page 383
10.2 Principal Components Analysis......Page 384
10.2.1 What Are Principal Components?......Page 385
10.2.2 Another Interpretation of Principal Components......Page 389
Scaling the Variables......Page 390
The Proportion of Variance Explained......Page 392
Deciding How Many Principal Components to Use......Page 393
10.2.4 Other Uses for Principal Components......Page 394
10.3 Clustering Methods......Page 395
10.3.1 K-Means Clustering......Page 396
10.3.2 Hierarchical Clustering......Page 400
Interpreting a Dendrogram......Page 401
The Hierarchical Clustering Algorithm......Page 404
Choice of Dissimilarity Measure......Page 406
Small Decisions with Big Consequences......Page 409
Other Considerations in Clustering......Page 410
10.4 Lab 1: Principal Components Analysis......Page 411
10.5.1 K-Means Clustering......Page 414
10.5.2 Hierarchical Clustering......Page 415
10.6.1 PCA on the NCI60 Data......Page 417
10.6.2 Clustering the Observations of the NCI60 Data......Page 419
10.7 Exercises......Page 423
 Index......Page 429