REGRESSION models, methods and applications.

Preface to the Second Edition
Preface to the First Edition
Contents
About the Authors
1 Introduction
	1.1 Examples of Applications
	1.2 First Steps
		1.2.1 Univariate Distributions of the Variables
		1.2.2 Graphical Association Analysis
	1.3 Notational Remarks
2 Regression Models
	2.1 Introduction
	2.2 Linear Regression Models
		2.2.1 Simple Linear Regression Model
		2.2.2 Multiple Linear Regression
	2.3 Regression with Binary Response Variables: The Logit Model
	2.4 Mixed Models
	2.5 Simple Nonparametric Regression
	2.6 Additive Models
	2.7 Generalized Additive Models
	2.8 Geoadditive Regression
	2.9 Regression Models with Functional Covariates
	2.10 Distributional Regression
		2.10.1 Regression Models for Location, Scale, and Shape
		2.10.2 Quantile Regression
		2.10.3 Hazard Regression Models
	2.11 Regression Models and Machine Learning
		2.11.1 Trees and Random Forests
		2.11.2 Neural Networks and Deep Learning
	2.12 Models in a Nutshell
		2.12.1 Linear Models (LMs, Chaps. 3摥映數爠eflinkkapitelspslinearemodelle33 and 4摥映數爠eflinkextensionslinearmodel44)
		2.12.2 Logit Model (Chap. 5摥映數爠eflinkkapitelspsgeneralisiertelinearemodelle55)
		2.12.3 Poisson Regression (Chap. 5摥映數爠eflinkkapitelspsgeneralisiertelinearemodelle55)
		2.12.4 Generalized Linear Models (GLMs, Chaps. 5摥映數爠eflinkkapitelspsgeneralisiertelinearemodelle55 and 6摥映數爠eflinkKategorialeRegressionsmodelle66)
		2.12.5 Linear Mixed Models (LMMs, Chap. 7摥映數爠eflinkGemischteModelle77)
		2.12.6 Additive Models and Extensions (AMs, Chaps. 8摥映數爠eflinkkapitelspsnichtparametrisch188 and 9摥映數爠eflinkkapitelspsadditiv99)
		2.12.7 Generalized Additive (Mixed) Models (GA(M)Ms, Chap. 9摥映數爠eflinkkapitelspsadditiv99)
		2.12.8 Structured Additive Regression (STAR, Chap. 9摥映數爠eflinkkapitelspsadditiv99)
		2.12.9 Quantile Regression (Chap. 10摥映數爠eflinkdistributionalregression1010)
		2.12.10 GAMLSS (Chap. 10摥映數爠eflinkdistributionalregression1010)
3 The Classical Linear Model
	3.1 Model Definition
		3.1.1 Model Parameters, Estimation, and Residuals
		3.1.2 Discussion of Model Assumptions
		3.1.3 Modeling the Effects of Covariates
	3.2 Parameter Estimation
		3.2.1 Estimation of Regression Coefficients
		3.2.2 Estimation of the Error Variance
		3.2.3 Properties of the Estimators
	3.3 Hypothesis Testing and Confidence Intervals
		3.3.1 Exact F-Test
		3.3.2 Confidence Regions and Prediction Intervals
	3.4 Model Choice and Variable Selection
		3.4.1 Bias, Variance and Prediction Quality
		3.4.2 Model Choice Criteria
		3.4.3 Practical Use of Model Choice Criteria
		3.4.4 Model Diagnosis
	3.5 Bibliographic Notes and Proofs
		3.5.1 Bibliographic Notes
		3.5.2 Proofs
4 Extensions of the Classical Linear Model
	4.1 The General Linear Model
		4.1.1 Model Definition
		4.1.2 Weighted Least Squares
		4.1.3 Heteroscedastic Errors
		4.1.4 Autocorrelated Errors
	4.2 Regularization Techniques
		4.2.1 Statistical Regularization
		4.2.2 Ridge Regression
		4.2.3 Least Absolute Shrinkage and Selection Operator
		4.2.4 Geometric Properties of Regularized Estimates
		4.2.5 Partial Regularization
	4.3 Boosting Linear Regression Models
		4.3.1 Basic Principles
		4.3.2 Componentwise Boosting
		4.3.3 Generic Componentwise Boosting
	4.4 Bayesian Linear Models
		4.4.1 Standard Conjugate Analysis
		4.4.2 Regularization Priors
		4.4.3 Classical Bayesian Model Choice (and Beyond)
		4.4.4 Spike and Slab Priors
	4.5 Bibliographic Notes and Proofs
		4.5.1 Bibliographic Notes
		4.5.2 Proofs
5 Generalized Linear Models
	5.1 Binary Regression
		5.1.1 Binary Regression Models
		5.1.2 Maximum Likelihood Estimation
		5.1.3 Testing Linear Hypotheses
		5.1.4 Criteria for Model Fit and Model Choice
		5.1.5 Estimation of the Overdispersion Parameter
	5.2 Count Data Regression
		5.2.1 Models for Count Data
		5.2.2 Estimation and Testing: Likelihood Inference
		5.2.3 Criteria for Model Fit and Model Choice
		5.2.4 Estimation of the Overdispersion Parameter
	5.3 Models for Nonnegative Continuous Response Variables
	5.4 Generalized Linear Models
		5.4.1 General Model Definition
		5.4.2 Likelihood Inference
	5.5 Quasi-likelihood Models
	5.6 Bayesian Generalized Linear Models
		5.6.1 Posterior Mode Estimation
		5.6.2 Fully Bayesian Inference via MCMC Simulation Techniques
		5.6.3 MCMC-Based Inference Using Data Augmentation
	5.7 Regularization and Boosting in Generalized Linear Models
	5.8 Bibliographic Notes and Proofs
		5.8.1 Bibliographic Notes
		5.8.2 Proofs
6 Categorical Regression Models
	6.1 Introduction
	6.2 Models for Unordered Categories
	6.3 Ordinal Models
		6.3.1 The Cumulative Model
		6.3.2 The Sequential Model
	6.4 Estimation and Testing: Likelihood Inference
	6.5 Bibliographic Notes
7 Mixed Models
	7.1 Linear Mixed Models for Longitudinal and Clustered Data
		7.1.1 Random Intercept Models
		7.1.2 Random Coefficient or Slope Models
		7.1.3 General Model Definition and Matrix Notation
		7.1.4 Conditional and Marginal Formulation
		7.1.5 Stochastic Covariates
	7.2 General Linear Mixed Models
	7.3 Likelihood Inference in LMMs
		7.3.1 Known Variance–Covariance Parameters
		7.3.2 Unknown Variance–Covariance Parameters
		7.3.3 Variability of Fixed and Random Effects Estimators
		7.3.4 Testing Hypotheses
	7.4 Bayesian Linear Mixed Models
		7.4.1 Estimation for Known Covariance Structure
		7.4.2 Estimation for Unknown Covariance Structure
	7.5 Generalized Linear Mixed Models
		7.5.1 GLMMs for Longitudinal and Clustered Data
		7.5.2 Conditional and Marginal Models
		7.5.3 GLMMs in General Form
	7.6 Likelihood and Bayesian Inference in GLMMs
		7.6.1 Penalized Likelihood and Empirical Bayes Estimation
		7.6.2 Fully Bayesian Inference Using MCMC
	7.7 Practical Application of Mixed Models
		7.7.1 General Guidelines and Recommendations
		7.7.2 Case Study on Sales of Orange Juice
	7.8 Bibliographic Notes and Proofs
		7.8.1 Bibliographic Notes
		7.8.2 Proofs
8 Nonparametric Regression
	8.1 Univariate Smoothing
		8.1.1 Polynomial Splines
		8.1.2 Penalized Splines (P-Splines)
		8.1.3 General Penalization Approaches
		8.1.4 Smoothing Splines
		8.1.5 Random Walks and State-Space Models
		8.1.6 Kriging
		8.1.7 Local Smoothing Procedures
		8.1.8 General Scatter Plot Smoothing
		8.1.9 Choosing the Smoothing Parameter
		8.1.10 Adaptive Smoothing Approaches
	8.2 Bivariate and Spatial Smoothing
		8.2.1 Tensor Product P-Splines
		8.2.2 Radial Basis Functions and Thin Plate Splines
		8.2.3 Kriging: Spatial Smoothing with Continuous Location Variables
		8.2.4 Markov Random Fields: Spatial Smoothing with Discrete Location Variables
		8.2.5 Summary of Roughness Penalty Approaches
		8.2.6 Local and Adaptive Smoothing
	8.3 Higher-Dimensional Smoothing
	8.4 Bibliographic Notes
9 Structured Additive Regression
	9.1 Additive Models
	9.2 Geoadditive Regression
	9.3 Models with Interactions
		9.3.1 Varying Coefficient Models
		9.3.2 Interactions Between Two Continuous Covariates
	9.4 Additive Mixed Models
	9.5 Structured Additive Regression
	9.6 Inference
		9.6.1 Penalized Least Squares and Penalized Likelihood Estimation
		9.6.2 Empirical Bayes Inference Based on Mixed Model Representation
		9.6.3 Fully Bayesian Inference Based on MCMC
		9.6.4 Boosting STAR Models
	9.7 Case Study: Malnutrition in Zambia
		9.7.1 General Guidelines
		9.7.2 Descriptive Analysis
		9.7.3 Modeling Variants
		9.7.4 Estimation Results and Model Evaluation
		9.7.5 Automatic Function Selection
	9.8 Bibliographic Notes
10 Distributional Regression Models
	10.1 Quantile Regression
		10.1.1 Quantiles
		10.1.2 Linear Quantile Regression
		10.1.3 Bayesian Quantile Regression
		10.1.4 Additive Quantile Regression
	10.2 Generalized Additive Models for Location, Scale and Shape
		10.2.1 Heteroscedastic Normal Models
		10.2.2 General Model Specification
		10.2.3 Inference
		10.2.4 Multivariate GAMLSS and Copula Regression
	10.3 Other Distributional Regression Approaches
		10.3.1 Expectile Regression
		10.3.2 Conditional Transformation Models
		10.3.3 Hazard Regression Models
	10.4 Bibliographic Notes and Proofs
		10.4.1 Bibliographic Notes
		10.4.2 Proofs
A Matrix Algebra
	A.1  Definition and Elementary Matrix Operations
	A.2  Rank of a Matrix
	A.3  Block Matrices and the Matrix Inversion Lemma
	A.4  Determinant and Trace of a Matrix
	A.5  Generalized Inverse
	A.6  Eigenvalues and Eigenvectors
	A.7  Quadratic Forms
	A.8  Differentiation of Matrix Functions
Appendix B Probability Calculus and Statistical Inference
	B.1  Some Univariate Distributions
	B.2  Random Vectors
	B.3  Multivariate Normal Distribution
		B.3.1  Definition and Properties
		B.3.2  The Singular Multivariate Normal Distribution
		B.3.3  Distributions of Quadratic Forms
		B.3.4  Multivariate t-Distribution
		B.3.5  Normal-Inverse Gamma Distribution
	B.4  Likelihood Inference
		B.4.1  Maximum Likelihood Estimation
		B.4.2  Numerical Computation of the MLE
		B.4.3  Asymptotic Properties of the MLE
		B.4.4  Likelihood-Based Tests of Linear Hypotheses
		B.4.5  Model Choice
	B.5  Bayesian Inference
		B.5.1  Basic Concepts of Bayesian Inference
		B.5.2  Point and Interval Estimation
		B.5.3  MCMC Methods
		B.5.4  Model Selection
		B.5.5  Model Averaging
Index