Statistical Learning From A Regression Perspective

Preface to the Third Edition
Preface to the Second Edition
Preface to the First Edition
	Endnotes
	References
Contents
1 Statistical Learning as a Regression Problem
	1.1 Getting Started
	1.2 Setting the Regression Context
	1.3 Revisiting the Ubiquitous Linear Regression Model
		1.3.1 Problems in Practice
	1.4 Working with Statistical Models that are Wrong
		1.4.1 An Alternative Approach to Regression
		1.4.2 More on Statistical Inference with Wrong Models
		1.4.3 Introduction to Sandwich Standard Errors
		1.4.4 Introduction to Conformal Inference
		1.4.5 Introduction to the Nonparametric Bootstrap
		1.4.6 Wrong Regression Models with Binary Response Variables
	1.5 The Transition to Statistical Learning
		1.5.1 Models Versus Algorithms
	1.6 Some Initial Concepts
		1.6.1 Overall Goals of Statistical Learning
		1.6.2 Forecasting with Supervised Statistical Learning
		1.6.3 Overfitting
		1.6.4 Data Snooping
		1.6.5 Some Constructive Responses to Overfitting and Data Snooping
			1.6.5.1 Training Data, Evaluation Data, and Test Data
			1.6.5.2 Data Splitting
			1.6.5.3 Cross-Validation
		1.6.6 Loss Functions and Related Concepts
			1.6.6.1 Definitions of In-Sample and Out-Of-Sample Performance
			1.6.6.2 Categorical Response Variables
			1.6.6.3 Asymmetric Loss
		1.6.7 The Bias–Variance Tradeoff
		1.6.8 Linear Estimators
		1.6.9 Degrees of Freedom
		1.6.10 Basis Functions
		1.6.11 The Curse of Dimensionality
	1.7 Statistical Learning in Context
	Demonstrations and Exercises
		Set 1
		Set 2
		Set 3
	Endnotes
	Endnotes
	References
2 Splines, Smoothers, and Kernels
	2.1 Introduction
	2.2 Regression Splines
		2.2.1 Piecewise Linear Population Approximations
		2.2.2 Polynomial Regression Splines
		2.2.3 Natural Cubic Splines
		2.2.4 B-Splines
	2.3 Penalized Smoothing
		2.3.1 Shrinkage and Regularization
			2.3.1.1 Ridge Regression
			2.3.1.2 A Ridge Regression Illustration
			2.3.1.3 The Least Absolute Shrinkage and Selection Operator (LASSO)
			2.3.1.4 A Lasso Regression Illustration
	2.4 Penalized Regression Splines
		2.4.1 An Application
	2.5 Smoothing Splines
		2.5.1 A Smoothing Splines Illustration
	2.6 Locally Weighted Regression as a Smoother
		2.6.1 Nearest Neighbor Methods
		2.6.2 Locally Weighted Regression
			2.6.2.1 A Lowess Illustration
	2.7 Smoothers for Multiple Predictors
		2.7.1 Smoothing in Two Dimensions
		2.7.2 The Generalized Additive Model
			2.7.2.1 A GAM Fitting Algorithm
			2.7.2.2 An Illustration Using the Regression Splines Implementation of the Generalized Additive Model
	2.8 Smoothers with Categorical Variables
		2.8.1 An Illustration Using the Generalized Additive Model with a Binary Outcome
	2.9 An Illustration of Statistical Inference After Model Selection
		2.9.1 Level I Versus Level II Summary
	2.10 Kernelized Regression
		2.10.1 Radial Basis Kernel
		2.10.2 ANOVA Radial Basis Kernel
		2.10.3 A Kernel Regression Application
	2.11 Summary and Conclusions
	Endnotes
	Endnotes
	References
3 Classification and Regression Trees (CART)
	3.1 Introduction
	3.2 An Introduction to Recursive Partitioning in CART
	3.3 The Basic Ideas in More Depth
		3.3.1 Tree Diagrams for Showing What the Greedy Algorithm Determined
		3.3.2 An Initial Application
		3.3.3 Classification and Forecasting with CART
		3.3.4 Confusion Tables
		3.3.5 CART as an Adaptive Nearest Neighbor Method
	3.4 The Formalities of Splitting a Node
	3.5 An Illustrative Prison Inmate Risk Assessment Using CART
	3.6 Classification Errors and Costs
		3.6.1 Default Costs in CART
		3.6.2 Prior Probabilities and Relative MisclassificationCosts
	3.7 Varying the Prior and the Complexity Parameter
	3.8 An Example with Three Response Categories
	3.9 Regression Trees
		3.9.1 A CART Application for the Correlates of a Student's GPA in High School
	3.10 Pruning
	3.11 Missing Data
		3.11.1 Missing Data with CART
	3.12 More on CART Instability
	3.13 Summary of Statistical Inference with CART
		3.13.1 Summary of Statistical Inference for CART Forecasts
	3.14 Overall Summary and Conclusions
	Exercises
	Exercises
		Problem Set 1
		Problem Set 2
		Problem Set 3
		Problem Set 4
	Endnotes
	References
4 Bagging
	4.1 Introduction
	4.2 The Bagging Algorithm
	4.3 Some Bagging Details
		4.3.1 Revisiting the CART Instability Problem
		4.3.2 Resampling Methods for Bagging
		4.3.3 Votes Over Trees and Probabilities
		4.3.4 Forecasting and Imputation
		4.3.5 Bagging Estimation and Statistical Inference
		4.3.6 Margins for Classification
		4.3.7 Using Out-of-Bag Observations as Test Data
		4.3.8 Bagging and Bias
	4.4 Some Limitations of Bagging
		4.4.1 Sometimes Bagging Cannot Help
		4.4.2 Sometimes Bagging Can Make the Estimation Bias Worse
		4.4.3 Sometimes Bagging Can Make the Estimation Variance Worse
	4.5 A Bagging Illustration
	4.6 Summary and Conclusions
	Exercises
	Exercises
		Problem Set 1
		Problem Set 2
		Problem Set 3
		Problem Set 4
	Endnotes
	References
5 Random Forests
	5.1 Introduction and Overview
		5.1.1 Unpacking How Random Forests Works
	5.2 An Initial Random Forests Illustration
	5.3 A Few Technical Formalities
		5.3.1 What Is a Random Forest?
		5.3.2 Margins and Generalization Error for Classifiers in General
		5.3.3 Generalization Error for Random Forests
		5.3.4 The Strength of a Random Forest
		5.3.5 Dependence
		5.3.6 Putting It Together
			5.3.6.1 Benefits From Interpolation
			5.3.6.2 Benefits from Averaging
			5.3.6.3 Defeating Competition Between Predictors
			5.3.6.4 All Together Now
	5.4 Random Forests and Adaptive Nearest Neighbor Methods
	5.5 Introducing Misclassification Costs
		5.5.1 A Brief Illustration Using Asymmetric Costs
	5.6 Determining the Importance of the Predictors
		5.6.1 Contributions to the Fit
		5.6.2 Contributions to Prediction
			5.6.2.1 Some Examples of Importance Plots with Extensions
	5.7 Input Response Functions
		5.7.1 Partial Dependence Plot Example
		5.7.2 More than Two Response Classes
	5.8 Classification and the Proximity Matrix
		5.8.1 Clustering by Proximity Values
			5.8.1.1 Using Proximity Values to Impute Missing Data
			5.8.1.2 Using Proximities to Detect Outliers
	5.9 Empirical Margins
	5.10 Quantitative Response Variables
	5.11 A Random Forest Illustration Using a Quantitative Response Variable
	5.12 Statistical Inference with Random Forests
	5.13 Software and Tuning Parameters
	5.14 Bayesian Additive Regression Trees (BART)
	5.15 Summary and Conclusions
	Exercises
	Exercises
		Problem Set 1
		Problem Set 2
		Problem Set 3
	Endnotes
	References
6 Boosting
	6.1 Introduction
	6.2 AdaBoost
		6.2.1 A Toy Numerical Example of AdaBoost.M1
		6.2.2 Why Does Boosting Work so Well for Classification?
			6.2.2.1 Boosting as a Margin Maximizer
			6.2.2.2 Boosting as a Statistical Optimizer
			6.2.2.3 Boosting as an Interpolator
	6.3 Stochastic Gradient Boosting
		6.3.1 Gradient Boosting More Formally
		6.3.2 Stochastic Gradient Boosting in Practice
		6.3.3 Tuning Parameters
		6.3.4 Output
	6.4 Asymmetric Costs
	6.5 Boosting, Estimation, and Consistency
	6.6 A Binomial Example
	6.7 Boosting for Statistical Inference and Forecasting
		6.7.1 An Imputation Example
	6.8 A Quantile Regression Example
	6.9 Boosting in Service of Causal Inference in Observational Studies
	6.10 Summary and Conclusions
	Exercises
	Exercises
		Problem Set 1
		Problem Set 2
		Problem Set 3
	Endnotes
	References
7 Support Vector Machines
	7.1 Introduction
	7.2 Support Vector Machines in Pictures
		7.2.1 The Support Vector Classifier
		7.2.2 Support Vector Machines
	7.3 Support Vector Machines More Formally
		7.3.1 The Support Vector Classifier Again: The SeparableCase
		7.3.2 The Nonseparable Case
		7.3.3 Support Vector Machines
		7.3.4 SVM for Regression
		7.3.5 Statistical Inference for Support Vector Machines
	7.4 A Classification Example
	7.5 Summary and Conclusions
	Exercises
	Exercises
		Problem Set 1
		Problem Set 2
		Problem Set 3
	Endnotes
	References
8 Neural Networks
	8.1 Introduction
	8.2 Conventional (Vanilla) Neural Networks
		8.2.1 Implementation of Gradient Descent
		8.2.2 Statistical Inference with Neural Networks
		8.2.3 An Application
		8.2.4 Some Recent Developments
		8.2.5 Implications of Conventional Neural Nets for Practice
	8.3 Deep Learning with Neural Networks
		8.3.1 Convolutional Neural Networks
			8.3.1.1 CNNs in More Detail
		8.3.2 Recurrent Neural Networks
			8.3.2.1 Statistical Inference for RNNs
			8.3.2.2 RNN Software
			8.3.2.3 An RNN Application
		8.3.3 Adversarial Neural Networks
	8.4 Conclusions
	Demonstrations and Exercises
		Set 1
		Set 2
	Endnotes
	References
9 Reinforcement Learning and Genetic Algorithms
	9.1 Introduction to Reinforcement Learning
	9.2 Genetic Algorithms
	9.3 An Application
	9.4 Conclusions
	Demonstrations and Exercises
		Set 1
		Set 2
	Endnotes
	References
10 Integrating Themes and a Bit of Craft Lore
	10.1 Some Integrating Technical Themes
	10.2 Integrating Themes Addressing Ethics and Politics
	10.3 Some Suggestions for Day-to-Day Practice
		10.3.1 Choose the Right Data Analysis Procedure
		10.3.2 Get to Know Your Software
		10.3.3 Do Not Forget the Basics
		10.3.4 Getting Good Data
		10.3.5 Match Your Goals to What You Can Credibly Do
	10.4 Some Concluding Observations
	Endnotes
	References
Bibliography
Index