Linear Regression Analysis: Theory and Computing

Contents......Page 10
Preface......Page 6
List of Figures......Page 16
List of Tables......Page 18
1.1 Regression Model......Page 22
1.2 Goals of Regression Analysis......Page 25
1.3 Statistical Computing in Regression Analysis......Page 26
2.1 Introduction......Page 30
2.2 Least Squares Estimation......Page 31
2.3 Statistical Properties of the Least Squares Estimation......Page 34
2.4 Maximum Likelihood Estimation......Page 39
2.5 Confidence Interval on Regression Mean and Regression Prediction......Page 40
2.6 Statistical Inference on Regression Parameters......Page 42
2.7 Residual Analysis and Model Diagnosis......Page 46
2.8 Example......Page 49
3.1.1 Vector Space......Page 62
3.1.3 Dot Product and Projection......Page 65
3.2 Matrix Form of Multiple Linear Regression......Page 69
3.3 Quadratic Form of Random Variables......Page 70
3.4 Idempotent Matrices......Page 71
3.5 Multivariate Normal Distribution......Page 75
3.6 Quadratic Form of the Multivariate Normal Variables......Page 77
3.7 Least Squares Estimates of the Multiple Regression Parameters......Page 79
3.8 Matrix Form of the Simple Linear Regression......Page 83
3.9 Test for Full Model and Reduced Model......Page 85
3.10 Test for General Linear Hypothesis......Page 87
3.11 The Least Squares Estimates of Multiple Regression Parameters Under Linear Restrictions......Page 88
3.12 Confidence Intervals of Mean and Prediction in Multiple Regression......Page 90
3.13 Simultaneous Test for Regression Parameters......Page 91
3.14 Bonferroni Confidence Region for Regression Parameters......Page 92
3.15 Interaction and Confounding......Page 93
3.15.1 Interaction......Page 94
3.15.2 Confounding......Page 96
3.16 Regression with Dummy Variables......Page 98
3.17.1 Collinearity......Page 102
3.17.2 Variance Inflation......Page 106
3.18 Linear Model in Centered Form......Page 108
3.19.1 Orthogonalization......Page 113
3.19.2 QR Decomposition and LSE......Page 115
3.20.1 Purpose of the Residual Analysis......Page 117
3.20.2 Residual Plot......Page 118
3.20.4 PRESS Residual......Page 124
3.20.5 Identify Outlier Using PRESS Residual......Page 127
3.20.6 Test for Mean Shift Outlier......Page 129
3.22 Example......Page 136
4. Detection of Outliers and Inuential Observations in Multiple Linear Regression......Page 150
4.1.1 Simple Criteria for Model Comparison......Page 151
4.1.2 Bias in Error Estimate from Under-specified Model......Page 152
4.1.3 Cross Validation......Page 153
4.2 Detection of Outliers in Multiple Linear Regression......Page 154
4.3.1 Inuential Observation......Page 155
4.3.2 Notes on Outlier and Inuential Observation......Page 157
4.3.3 Residual Mean Square Error for Over-fitted Regression Model......Page 158
4.4 Test for Mean-shift Outliers......Page 160
4.5.1 Partial Residual Plot......Page 163
4.5.2 Component-plus-residual Plot......Page 167
4.6 Test for Inferential Observations......Page 168
4.7 Example......Page 171
5.1 Effect of Underfitting and Overfitting......Page 178
5.2.1 Some Naive Criteria......Page 186
5.2.2 PRESS and GCV......Page 187
5.2.3 Mallow\'s CP......Page 188
5.2.4 AIC, AICC, and BIC......Page 190
5.3.1 Backward Elimination......Page 192
5.3.3 Stepwise Search......Page 193
5.4 Examples......Page 194
5.5 Other Related Issues......Page 200
5.5.1 Variance Importance or Relevance......Page 201
5.5.2 PCA and SIR......Page 207
6. Model Diagnostics......Page 216
6.1.1 Heteroscedasticity......Page 218
6.1.2 Likelihood Ratio Test, Wald, and Lagrange Multiplier Test......Page 219
6.1.3.1 White\'s Test......Page 222
6.1.3.2 Park, Glesjer, and Breusch-Pagan-Godfrey Tests......Page 223
6.1.3.3 Goldfeld-Quandt test......Page 224
6.2 Detection of Regression Functional Form......Page 225
6.2.1 Box-Cox Power Transformation......Page 226
6.2.2 Additive Models......Page 228
6.2.3 ACE and AVAS......Page 231
6.2.4 Example......Page 232
7.1 Non-Full-Rank Linear Regression Models......Page 240
7.1.1 Generalized Inverse......Page 242
7.1.2 Statistical Inference on Null-Full-Rank Regression Models......Page 244
7.2 Generalized Least Squares......Page 250
7.2.1 Estimation of (β, σ2)......Page 251
7.2.2 Statistical Inference......Page 252
7.2.3 Misspecification of the Error Variance Structure......Page 253
7.2.4 Typical Error Variance Structures......Page 254
7.2.5 Example......Page 257
7.3 Ridge Regression and LASSO......Page 259
7.3.1 Ridge Shrinkage Estimator......Page 260
7.3.2 Connection with PCA......Page 264
7.3.3 LASSO and Other Extensions......Page 267
7.3.4 Example......Page 271
7.4 Parametric Nonlinear Regression......Page 280
7.4.1 Least Squares Estimation in Nonlinear Regression......Page 282
7.4.2 Example......Page 284
8.1 Introduction: A Motivating Example......Page 290
8.2.1 Exponential Family......Page 293
8.2.2 Linear Predictor and Link Functions......Page 294
8.3.1 Likelihood Equations......Page 295
8.3.2 Fisher’s Information Matrix......Page 296
8.3.3 Optimization of the Likelihood......Page 297
8.4.1 Wald, Likelihood Ratio, and Score Test......Page 299
8.4.2 Other Model Fitting Issues......Page 302
8.5.1 Interpreting the Logistic Model......Page 303
8.5.2 Estimation of the Logistic Model......Page 305
8.5.3 Example......Page 306
8.6.1 The Loglinear Model......Page 308
8.6.2 Example......Page 309
9.1.1 Bayesian Inference in General......Page 318
9.1 Bayesian Linear Models......Page 14
9.1.2 Conjugate Normal-Gamma Priors......Page 320
9.1.3 Inference in Bayesian Linear Model......Page 323
9.1.4 Bayesian Inference via MCMC......Page 324
9.1.5 Prediction......Page 327
9.1.6 Example......Page 328
9.2 Bayesian Model Averaging......Page 330
Bibliography......Page 338
Index......Page 346