Bayesian Modeling and Computation in Python

Cover
Half Title
Series Page
Title Page
Copyright Page
Dedication
Contents
Foreword
Preface
Symbols
1. Bayesian Inference
	1.1. Bayesian Modeling
		1.1.1. Bayesian Models
		1.1.2. Bayesian Inference
	1.2. A DIY Sampler, Do Not Try This at Home
	1.3. Say Yes to Automating Inference, Say No to Automated Model Building
	1.4. A Few Options to Quantify Your Prior Information
		1.4.1. Conjugate Priors
		1.4.2. Objective Priors
		1.4.3. Maximum Entropy Priors
		1.4.4. Weakly Informative Priors and Regularization Priors
		1.4.5. Using Prior Predictive Distributions to Assess Priors
	1.5. Exercises
2. Exploratory Analysis of Bayesian Models
	2.1. There is Life After Inference, and Before Too!
	2.2. Understanding Your Assumptions
	2.3. Understanding Your Predictions
	2.4. Diagnosing Numerical Inference
		2.4.1. Effective Sample Size
		2.4.2. Potential Scale Reduction Factor R
		2.4.3. Monte Carlo Standard Error
		2.4.4. Trace Plots
		2.4.5. Autocorrelation Plots
		2.4.6. Rank Plots
		2.4.7. Divergences
		2.4.8. Sampler Parameters and Other Diagnostics
	2.5. Model Comparison
		2.5.1. Cross-validation and LOO
		2.5.2. Expected Log Predictive Density
		2.5.3. Pareto Shape Parameter, k is Large
		2.5.4. Interpreting p_loo When Pareto k
		2.5.5. LOO-PIT
		2.5.6. Model Averaging
	2.6. Exercises
3. Linear Models and Probabilistic Programming Languages
	3.1. Comparing Two (or More) Groups
		3.1.1. Comparing Two PPLs
	3.2. Linear Regression
		3.2.1. Linear Penguins
		3.2.2. Predictions
		3.2.3. Centering
	3.3. Multiple Linear Regression
		3.3.1. Counterfactuals
	3.4. Generalized Linear Models
		3.4.1. Logistic Regression
		3.4.2. Classifying Penguins
		3.4.3. Interpreting Log Odds
	3.5. Picking Priors in Regression Models
	3.6. Exercises
4. Extending Linear Models
	4.1. Transforming Covariates
	4.2. Varying Uncertainty
	4.3. Interaction Effects
	4.4. Robust Regression
	4.5. Pooling, Multilevel Models, and Mixed Effects
		4.5.1. Unpooled Parameters
		4.5.2. Pooled Parameters
		4.5.3. Mixing Group and Common Parameters
	4.6. Hierarchical Models
		4.6.1. Posterior Geometry Matters
		4.6.2. Predictions at Multiple Levels
		4.6.3. Priors for Multilevel Models
	4.7. Exercises
5. Splines
	5.1. Polynomial Regression
	5.2. Expanding the Feature Space
	5.3. Introducing Splines
	5.4. Building the Design Matrix using Patsy
	5.5. Fitting Splines in PyMC3
	5.6. Choosing Knots and Prior for Splines
		5.6.1. Regularizing Prior for Splines
	5.7. Modeling CO2 Uptake with Splines
	5.8. Exercises
6. Time Series
	6.1. An Overview of Time Series Problems
	6.2. Time Series Analysis as a Regression Problem
		6.2.1. Design Matrices for Time Series
		6.2.2. Basis Functions and Generalized Additive Model
	6.3. Autoregressive Models
		6.3.1. Latent AR Process and Smoothing
		6.3.2. (S)AR(I)MA(X)
	6.4. State Space Models
		6.4.1. Linear Gaussian State Space Models and Kalman filter
		6.4.2. ARIMA, Expressed as a State Space Model
		6.4.3. Bayesian Structural Time Series
	6.5. Other Time Series Models
	6.6. Model Criticism and Choosing Priors
		6.6.1. Priors for Time Series Models
	6.7. Exercises
7. Bayesian Additive Regression Trees
	7.1. Decision Trees
		7.1.1. Ensembles of Decision Trees
	7.2. The BART Model
	7.3. Priors for BART
		7.3.1. Prior Independence
		7.3.2. Prior for the Tree Structure Tj
		7.3.3. Prior for the Leaf Values uij and Number of Trees m
	7.4. Fitting Bayesian Additive Regression Trees
	7.5. BART Bikes
	7.6. Generalized BART Models
	7.7. Interpretability of BARTs
		7.7.1. Partial Dependence Plots
		7.7.2. Individual Conditional Expectation
	7.8. Variable Selection
	7.9. Priors for BART in PyMC3
	7.10. Exercises
8. Approximate Bayesian Computation
	8.1. Life Beyond Likelihood
	8.2. Approximating the Approximated Posterior
	8.3. Fitting a Gaussian the ABC-way
	8.4. Choosing the Distance Function,  and the Summary Statistics
		8.4.1. Choosing the Distance
		8.4.2. Choosing e
		8.4.3. Choosing Summary Statistics
	8.5. g-and-k Distribution
	8.6. Approximating Moving Averages
	8.7. Model Comparison in the ABC Context
		8.7.1. Marginal Likelihood and LOO
		8.7.2. Model Choice via Random Forest
		8.7.3. Model Choice for MA Model
	8.8. Choosing Priors for ABC
	8.9. Exercises
9. End to End Bayesian Workflows
	9.1. Workflows, Contexts, and Questions
		9.1.1. Applied Example: Airlines Flight Delays Problem
	9.2. Getting Data
		9.2.1. Sample Surveys
		9.2.2. Experimental Design
		9.2.3. Observational Studies
		9.2.4. Missing Data
		9.2.5. Applied Example: Collecting Airline Flight Delays Data
	9.3. Making a Model and Probably More Than One
		9.3.1. Questions to Ask Before Building a Bayesian Model
		9.3.2. Applied Example: Picking Flight Delay Likelihoods
	9.4. Choosing Priors and Predictive Priors
		9.4.1. Applied Example: Picking Priors for Flight Delays Model
	9.5. Inference and Inference Diagnostics
		9.5.1. Applied Example: Running Inference on Flight Delays Models
	9.6. Posterior Plots
		9.6.1. Applied Example: Posterior of Flight Delays Models
	9.7. Evaluating Posterior Predictive Distributions
		9.7.1. Applied Example: Posterior Predictive Distributions of Flight Delays
	9.8. Model Comparison
		9.8.1. Applied Example: Model Comparison with LOO of Flight Delays
	9.9. Reward Functions and Decisions
		9.9.1. Applied Example: Making Decisions Based on Flight Delays Modeling Results
	9.10. Sharing the Results With a Particular Audience
		9.10.1. Reproducibility of Analysis Workflow
		9.10.2. Understanding the Audience
		9.10.3. Static Visual Aids
		9.10.4. Reproducible Computing Environments
		9.10.5. Applied Example: Presenting Flight Delay Conclusions
	9.11. Experimental Example: Comparing Between Two Groups
	9.12. Exercises
10. Probabilistic Programming Languages
	10.1. A Systems Engineering Perspective of a PPL
		10.1.1. Example: Rainier
	10.2. Posterior Computation
		10.2.1. Getting the Gradient
		10.2.2. Example: Near Real Time Inference
	10.3. Application Programming Interfaces
		10.3.1. Example: Stan and Slicstan
		10.3.2. Example: PyMC3 and PyMC4
	10.4. PPL Driven Transformations
		10.4.1. Log Probabilities
		10.4.2. Random Variables and Distributions Transformations
		10.4.3. Example: Sampling Comparison between Bounded and Unbounded Random Variables
	10.5. Operation Graphs and Automatic Reparameterization
	10.6. Effect handling
		10.6.1. Example: Effect Handling in TFP and Numpyro
	10.7. Base Language, Code Ecosystem, Modularity and Everything Else
	10.8. Designing a PPL
		10.8.1. Shape Handling in PPLs
	10.9. Takeaways for the Applied Bayesian Practitioner
	10.10. Exercises
11. Appendiceal Topics
	11.1. Probability Background
		11.1.1. Probability
		11.1.2. Conditional Probability
		11.1.3. Probability Distribution
		11.1.4. Discrete Random Variables and Distributions
		11.1.5. Continuous Random Variables and Distributions
		11.1.6. Joint, Conditional and Marginal Distributions
		11.1.7. Probability Integral Transform (PIT)
		11.1.8. Expectations
		11.1.9. Transformations
		11.1.10. Limits
		11.1.11. Markov Chains
	11.2. Entropy
	11.3. Kullback-Leibler Divergence
	11.4. Information Criterion
	11.5. LOO in Depth
	11.6. Jeffreys' Prior Derivation
		11.6.1. Jeffreys' Prior for the Binomial Likelihood in Terms of θ
		11.6.2. Jeffreys' Prior for the Binomial Likelihood in Terms of k
		11.6.3. Jeffreys' Posterior for the Binomial Likelihood
	11.7. Marginal Likelihood
		11.7.1. The Harmonic Mean Estimator
		11.7.2. Marginal Likelihood and Model Comparison
		11.7.3. Bayes Factor vs WAIC and LOO
	11.8. Moving out of Flatland
	11.9. Inference Methods
		11.9.1. Grid Method
		11.9.2. Metropolis-Hastings
		11.9.3. Hamiltonian Monte Carlo
		11.9.4. Sequential Monte Carlo
		11.9.5. Variational Inference
	11.10. Programming References
		11.10.1. Which Programming Language?
		11.10.2. Version Control
		11.10.3. Dependency Management and Package Repositories
		11.10.4. Environment Management
		11.10.5. Text Editor vs Integrated Development Environment vs Notebook
		11.10.6. The Specific Tools Used for this Book
Glossary
Bibliography
Index