Handbook of Bayesian Variable Selection

Cover
Half Title
Series Page
Title Page
Copyright Page
Dedication
Contents
Preface
Biography
List of Contributors
List of Symbols
I. Spike-and-Slab Priors
	1. Discrete Spike-and-Slab Priors: Models and Computational Aspects
		1.1. Introduction
		1.2. Spike-and-Slab Priors for Linear Regression Models
			1.2.1. Stochastic Search MCMC
			1.2.2. Prediction via Bayesian Model Averaging
		1.3. Spike-and-Slab Priors for Non-Gaussian Data
			1.3.1. Compositional Count Data
		1.4. Structured Spike-and-Slab Priors for Biomedical Studies
			1.4.1. Network Priors
			1.4.2. Spiked Nonparametric Priors
		1.5. Scalable Bayesian Variable Selection
			1.5.1. Variational Inference
		1.6. Conclusion
		Bibliography
	2. Recent Theoretical Advances with the Discrete Spike-and Slab Priors
		2.1. Introduction
		2.2. Optimal Recovery in Gaussian Sequence Models
			2.2.1. Minimax Rate in Nearly Black Gaussian Mean Models
			2.2.2. Optimal Bayesian Recovery in `q-norm
			2.2.3. Optimal Contraction Rate for Other Variants of Priors
			2.2.4. Slow Contraction Rate for Light-tailed Priors
		2.3. Sparse Linear Regression Model
			2.3.1. Prior Construction and Assumptions
			2.3.2. Compatibility Conditions on the Design Matrix
			2.3.3. Posterior Contraction Rate
			2.3.4. Variable Selection Consistency
			2.3.5. Variable Selection with Discrete Spike and Zellner's g-Priors
			2.3.6. Bernstein-von Mises Theorem for the Posterior Distribution
		2.4. Extension to Generalized Linear Models
			2.4.1. Construction of the GLM Family
			2.4.2. Clipped GLM and Connections to Regression Settings
			2.4.3. Construction of Sparsity Favoring Prior
			2.4.4. Assumptions on Data Generating Distribution and Prior
			2.4.5. Adaptive Rate-Optimal Posterior Contraction Rate in `1-norm
		2.5. Optimality Results for Variational Inference in Linear Regression Models
		2.6. Discussion
		Bibliography
	3. Theoretical and Computational Aspects of Continuous Spike-and-Slab Priors
		3.1. Introduction
		3.2. Variable Selection in Linear Models
		3.3. Continuous Spike-and-Slab Priors
			3.3.1. Shrinking and Diffusing Priors
			3.3.2. Spike-and-Slab LASSO
		3.4. Theoretical Properties
			3.4.1. Variable Selection Consistency
			3.4.2. Novel Insights
			3.4.3. Examples
		3.5. Computations
			3.5.1. Skinny Gibbs for Scalable Posterior Sampling
			3.5.2. Skinny Gibbs for Non-Normal Spike-and-Slab Priors
		3.6. Generalizations
		3.7. Conclusion
		Bibliography
	4. Spike-and-Slab Meets LASSO: A Review of the Spike-and-Slab LASSO
		4.1. Introduction
		4.2. Variable Selection in High-Dimensions: Frequentist and Bayesian Strategies
			4.2.1. Penalized Likelihood Approaches
			4.2.2. Spike-and-Slab Priors
		4.3. The Spike-and-Slab LASSO
			4.3.1. Prior Specification
			4.3.2. Selective Shrinkage and Self-Adaptivity to Sparsity
			4.3.3. The Spike-and-Slab LASSO in Action
		4.4. Computational Details
			4.4.1. Coordinate-wise Optimization
			4.4.2. Dynamic Posterior Exploration
			4.4.3. EM Implementation of the Spike-and-Slab LASSO
		4.5. Uncertainty Quanti cation
			4.5.1. Debiasing the Posterior Mode
			4.5.2. Posterior Sampling for the Spike-and-Slab LASSO
		4.6. Illustrations
			4.6.1. Example on Synthetic Data
			4.6.2. Bardet-Beidl Syndrome Gene Expression Study
		4.7. Methodological Extensions
		4.8. Theoretical Properties
		4.9. Discussion
		Bibliography
	5. Adaptive Computational Methods for Bayesian Variable Selection
		5.1. Introduction
			5.1.1. Some Reasons to be Cheerful
			5.1.2. Adaptive Monte Carlo Methods
		5.2. Some Adaptive Approaches to Bayesian Variable Selection
		5.3. Two Adaptive Algorithms
			5.3.1. Linear Regression
			5.3.2. Non-Gaussian Models
		5.4. Examples
			5.4.1. Simulated Example: Linear Regression
			5.4.2. Fine Mapping for Systemic Lupus Erythematosus
			5.4.3. Analysing Environmental DNA Data
		5.5. Discussion
		Bibliography
II. Continuous Shrinkage Priors
	6. Theoretical Guarantees for the Horseshoe and Other Global-Local Shrinkage Priors
		6.1. Introduction
			6.1.1. Model and Notation
			6.1.2. Global-Local Shrinkage Priors and Spike-and-Slab Priors
			6.1.3. Performance Measures
		6.2. Global-Local Shrinkage Priors
		6.3. Recovery Guarantees
			6.3.1. Non-Adaptive Posterior Concentration Theorems
			6.3.2. Proof Techniques
			6.3.3. Adaptive Posterior Concentration Theorems
			6.3.4. Other Sparsity Assumptions
			6.3.5. Implications for Practice
		6.4. Uncertainty Quanti cation Guarantees
			6.4.1. Credible Intervals
			6.4.2. Credible Balls
			6.4.3. Implications for Practice
		6.5. Variable Selection Guarantees
			6.5.1. Thresholding on the Amount of Shrinkage
			6.5.2. Checking for Zero in Marginal Credible Intervals
		6.6. Discussion
		Bibliography
	7. MCMC for Global-Local Shrinkage Priors in High-Dimensional Settings
		7.1. Introduction
		7.2. Global-Local Shrinkage Priors
		7.3. Posterior Sampling
			7.3.1. Sampling Structured High-Dimensional Gaussians
			7.3.2. Blocking can be Advantageous
			7.3.3. Geometric Convergence
		7.4. Approximate MCMC
		7.5. Conclusion
		Bibliography
	8. Variable Selection with Shrinkage Priors via Sparse Posterior Summaries
		8.1. Introduction
		8.2. Penalized Credible Region Selection
			8.2.1. Gaussian Prior
			8.2.2. Global-Local Shrinkage Priors
			8.2.3. Example: Simulation Studies
			8.2.4. Example: Mouse Gene Expression Real-time PCR
		8.3. Approaches Based on Other Posterior Summaries
		8.4. Model Selection for Logistic Regression
		8.5. Graphical Model Selection
		8.6. Confounder Selection
		8.7. Time-Varying Coefficients
		8.8. Discussion
		Bibliography
III. Extensions to Various Modeling Frameworks
	9. Bayesian Model Averaging in Causal Inference
		9.1. Introduction to Causal Inference
			9.1.1. Potential Outcomes, Estimands, and Identifying Assumptions
			9.1.2. Estimation Strategies Using Outcome Regression, Propensity Scores, or Both
			9.1.3. Why Use BMA for Causal Inference?
		9.2. Failure of Traditional Model Averaging for Causal Inference Problems
		9.3. Prior Distributions Tailored Towards Causal Estimation
			9.3.1. Bayesian Adjustment for Confounding Prior
			9.3.2. Related Prior Distributions that Link Treatment and Outcome Models
		9.4. Bayesian Estimation of Treatment Effects
			9.4.1. Outcome Model Based Estimation
			9.4.2. Incorporating the Propensity Score into the Outcome Model
			9.4.3. BMA Coupled with Traditional Frequentist Estimators
			9.4.4. Analysis of Volatile Compounds on Cholesterol Levels
		9.5. Assessment of Uncertainty
		9.6. Extensions to Shrinkage Priors and Nonlinear Regression
		9.7. Conclusion
		Bibliography
	10. Variable Selection for Hierarchically-Related Outcomes: Models and Algorithms
		10.1. Introduction
		10.2. Model Formulations, Computational Challenges and Tradeoffs
		10.3. Illustrations on Published Case Studies
			10.3.1. Modelling eQTL Signals across Multiple Tissues
			10.3.2. Modelling eQTL Hotspots under Different Experimental Conditions
		10.4. Discussion
		Bibliography
	11. Bayesian Variable Selection in Spatial Regression Models
		11.1. Introduction
		11.2. Spatial Regression
		11.3. Regression Coefficients as Spatial Processes
			11.3.1. Spatially-Varying Coe cient Model
			11.3.2. Scalar-on-Image Regression
		11.4. Sparse Spatial Processes
			11.4.1. Discrete Mixture Priors
			11.4.2. Continuous Shrinkage Priors
		11.5. Application to Microbial Fungi across US Households
		11.6. Discussion
		Bibliography
	12. Effect Selection and Regularization in Structured Additive Distributional Regression
		12.1. Introduction
		12.2. Structured Additive Distributional Regression
			12.2.1. Basic Model Structure
			12.2.2. Predictor Components
			12.2.3. Common Response Distributions
			12.2.4. Basic MCMC Algorithm
		12.3. Effect Selection Priors
			12.3.1. Challenges
			12.3.2. Spike-and-Slab Priors for Effect Selection
			12.3.3. Regularization Priors for Effect Selection
		12.4. Application: Childhood Undernutrition in India
			12.4.1. Data
			12.4.2. A Main Effects Location-Scale Model
			12.4.3. Decomposing an Interaction Surface
		12.5. Other Regularization Priors for Functional Effects
			12.5.1. Locally Adaptive Regularization
			12.5.2. Shrinkage towards a Functional Subspace
		12.6. Summary and Discussion
		Bibliography
	13. Sparse Bayesian State-Space and Time-Varying Parameter Models
		13.1. Introduction
		13.2. Univariate Time-Varying Parameter Models
			13.2.1. Motivation and Model Definition
			13.2.2. The Inverse Gamma Versus the Ridge Prior
			13.2.3. Gibbs Sampling in the Non-Centered Parametrization
		13.3. Continuous Shrinkage Priors for Sparse TVP Models
			13.3.1. From the Ridge Prior to Continuous Shrinkage Priors
			13.3.2. Efficient MCMC Inference
			13.3.3. Application to US Inflation Modelling
		13.4. Spike-and-Slab Priors for Sparse TVP Models
			13.4.1. From the Ridge prior to Spike-and-Slab Priors
			13.4.2. Model Space MCMC
			13.4.3. Application to US Inflation Modelling
		13.5. Extensions
			13.5.1. Including Stochastic Volatility
			13.5.2. Sparse TVP Models for Multivariate Time Series
			13.5.3. Non-Gaussian Outcomes
			13.5.4. Log Predictive Scores for Comparing Shrinkage Priors
			13.5.5. BMA Versus Continuous Shrinkage Priors
		13.6. Discussion
		Bibliography
	14. Bayesian Estimation of Single and Multiple Graphs
		14.1. Introduction
		14.2. Bayesian Approaches for Single Graph Estimation
			14.2.1. Background on Graphical Models
			14.2.2. Bayesian Priors for Undirected Networks
			14.2.3. Bayesian Priors for Directed Networks
			14.2.4. Bayesian Network Inference for Non-Gaussian Data
		14.3. Multiple Graphs with Shared Structure
			14.3.1. Likelihood
			14.3.2. Prior Formulation
			14.3.3. Simulation and Case Studies
			14.3.4. Related Work
		14.4. Multiple Graphs with Shared Edge Values
			14.4.1. Likelihood
			14.4.2. Prior Formulation
			14.4.3. Analysis of Neuroimaging Data
		14.5. Multiple DAGs and Other Multiple Graph Approaches
		14.6. Related Topics
		14.7. Discussion
		Bibliography
IV. Other Approaches to Bayesian Variable Selection
	15. Bayes Factors Based on g-Priors for Variable Selection
		15.1. Bayes Factors
		15.2. Variable Selection in the Gaussian Linear Model
			15.2.1. Objective Prior Specifications
			15.2.2. Numerical Issues
			15.2.3. BayesVarSel and Applications
			15.2.4. Sensitivity to Prior Inputs
		15.3. Variable Selection for Non-Gaussian Data
			15.3.1. glmBfp and Applications
		15.4. Conclusion
		Bibliography
	16. Balancing Sparsity and Power: Likelihoods, Priors, and Misspecification
		16.1. Introduction
		16.2. BMS in Regression Models
		16.3. Interpreting BMS Under Misspeci cation
		16.4. Priors
		16.5. Prior Elicitation and Robustness
		16.6. Validity of Model Selection Uncertainty
		16.7. Finite-Dimensional Results
		16.8. High-Dimensional Results
		16.9. Balancing Sparsity and Power
		16.10. Examples
			16.10.1. Salary
			16.10.2. Colon Cancer
			16.10.3. Survival Analysis of Serum Free Light Chain Data
		16.11. Discussion
		Bibliography
	17. Variable Selection and Interaction Detection with Bayesian Additive Regression Trees
		17.1. Introduction
		17.2. BART Overview
			17.2.1. Specification of the BART Regularization Prior
			17.2.2. Posterior Calculation and Information Extraction
		17.3. Model-Free Variable Selection with BART
			17.3.1. Variable Selection with the Boston Housing Data
		17.4. Model-Free Interaction Detection with BART
			17.4.1. Variable Selection and Interaction Detection with the Friedman Simulation Setup
			17.4.2. Interaction Detection with the Boston Housing Data
		17.5. A Utility Based Approach to Variable Selection using BART Inference
			17.5.1. Step 1: BART Inference
			17.5.2. Step 2: Subset Search
			17.5.3. Step 3: Uncertainty Assessment
		17.6. Conclusion
		Bibliography
	18. Variable Selection for Bayesian Decision Tree Ensembles
		18.1. Introduction
			18.1.1. Running Example
			18.1.2. Possible Strategies
		18.2. Bayesian Additive Regression Trees
			18.2.1. Decision Trees and their Priors
			18.2.2. The BART Model
		18.3. Variable Importance Scores
			18.3.1. Empirical Bayes and Variable Importance Scores
		18.4. Sparsity Inducing Priors on s
			18.4.1. The Uniform Prior on s
			18.4.2. The Dirichlet Prior
			18.4.3. The Spike-and-Forest Prior
			18.4.4. Finite Gibbs Priors
		18.5. An Illustration: The WIPP Dataset
		18.6. Extensions
			18.6.1. Interaction Detection
			18.6.2. Structure in Predictors
		18.7. Discussion
		Bibliography
	19. Stochastic Partitioning for Variable Selection in Multivariate Mixture of Regression Models
		19.1. Introduction
		19.2. Mixture of Univariate Regression Models
			19.2.1. Model Fitting
			19.2.2. Variable Selection
		19.3. Stochastic Partitioning for Multivariate Mixtures
			19.3.1. Model Formulation
			19.3.2. Prior Speci cation
			19.3.3. Model Fitting
			19.3.4. Posterior Inference
		19.4. spavs and Application
			19.4.1. Choice of Hyperparameters and Other Input Values
			19.4.2. Post-Processing of MCMC Output and Posterior Inference
		19.5. Discussion
		Bibliography
Index