Regression: Models, Methods and Applications

1 Introduction......Page 12
1.1 Examples of Applications......Page 15
1.2.1 Univariate Distributions of the Variables......Page 22
1.2.2 Graphical Association Analysis......Page 24
1.3 Notational Remarks......Page 30
2.1 Introduction......Page 31
2.2.1 Simple Linear Regression Model......Page 32
2.2.2 Multiple Linear Regression......Page 36
2.3 Regression with Binary Response Variables: The Logit Model......Page 43
2.4 Mixed Models......Page 48
2.5 Simple Nonparametric Regression......Page 54
2.6 Additive Models......Page 59
2.7 Generalized Additive Models......Page 62
2.8 Geoadditive Regression......Page 65
2.9 Beyond Mean Regression......Page 71
2.9.1 Regression Models for Location, Scale, and Shape......Page 72
2.9.2 Quantile Regression......Page 76
2.10.3 Poisson Regression (Chap.5)......Page 78
2.10.5 Linear Mixed Models (LMMs, Chap.7)......Page 79
2.10.7 Generalized Additive (Mixed) Models (GA(M)Ms, Chap.9)......Page 80
2.10.9 Quantile Regression (Chap.10)......Page 81
3.1 Model Definition......Page 84
3.1.1 Model Parameters, Estimation, and Residuals......Page 88
3.1.2 Discussion of Model Assumptions......Page 89
3.1.3 Modeling the Effects of Covariates......Page 97
3.2.1 Estimation of Regression Coefficients......Page 115
3.2.2 Estimation of the Error Variance......Page 119
3.2.3 Properties of the Estimators......Page 121
3.3 Hypothesis Testing and Confidence Intervals......Page 136
3.3.1 Exact F-Test......Page 139
3.3.2 Confidence Regions and Prediction Intervals......Page 147
3.4 Model Choice and Variable Selection......Page 150
3.4.1 Bias, Variance and Prediction Quality......Page 153
3.4.2 Model Choice Criteria......Page 157
3.4.3 Practical Use of Model Choice Criteria......Page 161
3.4.4 Model Diagnosis......Page 166
3.5.2 Proofs......Page 179
4.1.1 Model Definition......Page 187
4.1.2 Weighted Least Squares......Page 188
4.1.3 Heteroscedastic Errors......Page 192
4.1.4 Autocorrelated Errors......Page 201
4.2 Regularization Techniques......Page 211
4.2.1 Statistical Regularization......Page 212
4.2.2 Ridge Regression......Page 213
4.2.3 Least Absolute Shrinkage and Selection Operator......Page 218
4.2.4 Geometric Properties of Regularized Estimates......Page 221
4.2.5 Partial Regularization......Page 226
4.3.1 Basic Principles......Page 227
4.3.2 Componentwise Boosting......Page 228
4.3.3 Generic Componentwise Boosting......Page 232
4.4 Bayesian Linear Models......Page 235
4.4.1 Standard Conjugate Analysis......Page 237
4.4.2 Regularization Priors......Page 247
4.4.3 Classical Bayesian Model Choice (and Beyond)......Page 253
4.4.4 Spike and Slab Priors......Page 263
4.5.1 Bibliographic Notes......Page 267
4.5.2 Proofs......Page 268
5 Generalized Linear Models......Page 278
5.1.1 Binary Regression Models......Page 279
5.1.2 Maximum Likelihood Estimation......Page 288
5.1.3 Testing Linear Hypotheses......Page 294
5.1.4 Criteria for Model Fit and Model Choice......Page 296
5.1.5 Estimation of the Overdispersion Parameter......Page 301
5.2.1 Models for Count Data......Page 302
5.2.2 Estimation and Testing: Likelihood Inference......Page 304
5.2.4 Estimation of the Overdispersion Parameter......Page 306
5.3 Models for Nonnegative Continuous Response Variables......Page 307
5.4.1 General Model Definition......Page 310
5.4.2 Likelihood Inference......Page 315
5.5 Quasi-likelihood Models......Page 318
5.6 Bayesian Generalized Linear Models......Page 320
5.6.1 Posterior Mode Estimation......Page 322
5.6.2 Fully Bayesian Inference via MCMC Simulation Techniques......Page 323
5.6.3 MCMC-Based Inference Using Data Augmentation......Page 325
5.7 Boosting Generalized Linear Models......Page 328
5.8.1 Bibliographic Notes......Page 329
5.8.2 Proofs......Page 330
6.1 Introduction......Page 334
6.2 Models for Unordered Categories......Page 338
6.3.1 The Cumulative Model......Page 343
6.3.2 The Sequential Model......Page 346
6.4 Estimation and Testing: Likelihood Inference......Page 352
6.5 Bibliographic Notes......Page 356
7 Mixed Models......Page 357
7.1.1 Random Intercept Models......Page 358
7.1.2 Random Coefficient or Slope Models......Page 365
7.1.3 General Model Definition and Matrix Notation......Page 369
7.1.4 Conditional and Marginal Formulation......Page 373
7.1.5 Stochastic Covariates......Page 374
7.2 General Linear Mixed Models......Page 376
7.3.1 Known Variance–Covariance Parameters......Page 379
7.3.2 Unknown Variance–Covariance Parameters......Page 380
7.3.3 Variability of Fixed and Random Effects Estimators......Page 386
7.3.4 Testing Hypotheses......Page 388
7.4 Bayesian Linear Mixed Models......Page 391
7.4.1 Estimation for Known Covariance Structure......Page 392
7.4.2 Estimation for Unknown Covariance Structure......Page 393
7.5.1 GLMMs for Longitudinal and Clustered Data......Page 397
7.5.2 Conditional and Marginal Models......Page 400
7.6 Likelihood and Bayesian Inference in GLMMs......Page 402
7.6.1 Penalized Likelihood and Empirical Bayes Estimation......Page 403
7.6.2 Fully Bayesian Inference Using MCMC......Page 405
7.7.1 General Guidelines and Recommendations......Page 409
7.7.2 Case Study on Sales of Orange Juice......Page 411
7.8.1 Bibliographic Notes......Page 417
7.8.2 Proofs......Page 418
8 Nonparametric Regression......Page 421
8.1.1 Polynomial Splines......Page 423
8.1.2 Penalized Splines (P-Splines)......Page 439
8.1.3 General Penalization Approaches......Page 454
8.1.4 Smoothing Splines......Page 456
8.1.5 Random Walks......Page 460
8.1.6 Kriging......Page 461
8.1.7 Local Smoothing Procedures......Page 468
8.1.8 General Scatter Plot Smoothing......Page 476
8.1.9 Choosing the Smoothing Parameter......Page 486
8.1.10 Adaptive Smoothing Approaches......Page 498
8.2 Bivariate and Spatial Smoothing......Page 508
8.2.1 Tensor Product P-Splines......Page 511
8.2.2 Radial Basis Functions and Thin Plate Splines......Page 520
8.2.3 Kriging: Spatial Smoothing with Continuous Location Variables......Page 523
8.2.4 Markov Random Fields......Page 529
8.2.5 Summary of Roughness Penalty Approaches......Page 535
8.2.6 Local and Adaptive Smoothing......Page 537
8.3 Higher-Dimensional Smoothing......Page 538
8.4 Bibliographic Notes......Page 539
9 Structured Additive Regression......Page 542
9.1 Additive Models......Page 543
9.2 Geoadditive Regression......Page 547
9.3 Models with Interactions......Page 550
9.3.1 Models with Varying Coefficient Terms......Page 551
9.3.2 Interactions Between Two Continuous Covariates......Page 554
9.4 Models with Random Effects......Page 556
9.5 Structured Additive Regression......Page 560
9.6.1 Penalized Least Squares or Likelihood Estimation......Page 568
9.6.2 Inference Based on Mixed Model Representation......Page 573
9.6.3 Bayesian Inference Based on MCMC......Page 575
9.7 Boosting STAR Models......Page 580
9.8.1 General Guidelines......Page 583
9.8.2 Descriptive Analysis......Page 587
9.8.3 Modeling Variants......Page 590
9.8.4 Estimation Results and Model Evaluation......Page 591
9.8.5 Automatic Function Selection......Page 596
9.9 Bibliographic Notes......Page 601
10 Quantile Regression......Page 603
10.1 Quantiles......Page 605
10.2.1 Classical Quantile Regression......Page 607
10.2.2 Bayesian Quantile Regression......Page 615
10.3 Additive Quantile Regression......Page 618
10.4.1 Bibliographic Notes......Page 622
10.4.2 Proofs......Page 624
A.1 Definition and Elementary Matrix Operations......Page 627
A.2 Rank of a Matrix......Page 632
A.3 Block Matrices and the Matrix Inversion Lemma......Page 634
A.4 Determinant and Trace of a Matrix......Page 635
A.6 Eigenvalues and Eigenvectors......Page 637
A.7 Quadratic Forms......Page 639
A.8 Differentiation of Matrix Functions......Page 641
B.1 Some Univariate Distributions......Page 644
B.2 Random Vectors......Page 650
B.3.1 Definition and Properties......Page 653
B.3.2 The Singular Multivariate Normal Distribution......Page 655
B.3.4 Multivariate t-Distribution......Page 656
B.3.5 Normal-Inverse Gamma Distribution......Page 657
B.4.1 Maximum Likelihood Estimation......Page 658
B.4.2 Numerical Computation of the MLE......Page 665
B.4.4 Likelihood-Based Tests of Linear Hypotheses......Page 667
B.4.5 Model Choice......Page 669
B.5.1 Basic Concepts of Bayesian Inference......Page 670
B.5.2 Point and Interval Estimation......Page 674
B.5.3 MCMC Methods......Page 675
B.5.4 Model Selection......Page 681
B.5.5 Model Averaging......Page 684
Bibliography......Page 685
Index......Page 695