Doing Bayesian Data Analysis: A Tutorial Introduction with R and BUGS

Foreword......Page 1
Contents......Page 3
1.1 Real people can read this book......Page 13
1.2 Prerequisites......Page 14
1.3 The organization of this book......Page 15
1.3.2 Where’s the equivalent of traditional test X in this book?......Page 16
1.5 Acknowledgments......Page 17
I The Basics: Parameters, Probability, Bayes’ Rule, and R......Page 19
2 Introduction: Models we believe in......Page 21
2.1 Models of observations and models of beliefs......Page 22
2.1.1 Models have parameters......Page 23
2.2.1 Estimation of parameter values......Page 25
2.2.3 Model comparison......Page 26
2.3.2 Invoking R and using the command line......Page 27
2.3.3 A simple example of R in action......Page 28
2.3.4 Getting help in R......Page 29
2.3.5.2 Variable names in R......Page 30
2.4 Exercises......Page 31
3 What is this stuff called probability?......Page 33
3.1.1 Coin flips: Why you should care......Page 34
3.2.1.1 Simulating a long-run relative frequency......Page 35
3.2.1.2 Deriving a long-run relative frequency......Page 36
3.2.2.1 Calibrating a subjective belief by preferences......Page 37
3.3 Probability distributions......Page 38
3.3.2 Continuous distributions: Rendezvous with density......Page 39
3.3.2.1 Properties of probability density functions......Page 41
3.3.2.2 The normal probability density function......Page 42
3.3.3 Mean and variance of a distribution......Page 44
3.3.3.1 Mean as minimized variance......Page 45
3.3.5 Highest density interval (HDI)......Page 46
3.4 Two-way distributions......Page 47
3.4.1 Marginal probability......Page 48
3.4.2 Conditional probability......Page 50
3.4.3 Independence of attributes......Page 51
3.5.1 R code for Figure 3.1......Page 52
3.6 Exercises......Page 53
4 Bayes’ Rule......Page 4
4.1.1 Derived from definitions of conditional probability......Page 56
4.1.2 Intuited from a two-way discrete table......Page 57
4.2 Applied to models and data......Page 59
4.2.1 Data order invariance......Page 61
4.2.2 An example with coin flipping......Page 62
4.3.2 Prediction of data values......Page 64
4.3.3 Model comparison......Page 65
4.3.5.1 Holmesian deduction......Page 68
4.4.1 R code for Figure 4.1......Page 69
4.5 Exercises......Page 71
II All the Fundamentals Applied to Inferring a Binomial Proportion......Page 75
5 Inferring a Binomial Proportion via Exact Mathematical Analysis......Page 77
5.1 The likelihood function: Bernoulli distribution......Page 78
5.2 A description of beliefs: The beta distribution......Page 79
5.2.1 Specifying a beta prior......Page 80
5.2.2 The posterior beta......Page 82
5.3.1 Estimating the binomial proportion......Page 83
5.3.2 Predicting data......Page 84
5.3.3 Model comparison......Page 85
5.4 Summary: How to do Bayesian inference......Page 87
5.5.1 R code for Figure 5.2......Page 88
5.6 Exercises......Page 91
6 Inferring a Binomial Proportion via Grid Approximation......Page 95
6.2 Discretizing a continuous prior density......Page 96
6.2.1 Examples using discretized priors......Page 97
6.3 Estimation......Page 99
6.4 Prediction of subsequent data......Page 100
6.6 Summary......Page 101
6.7.1 R code for Figure 6.2 etc......Page 102
6.8 Exercises......Page 104
7 Inferring a Binomial Proportion via the Metropolis Algorithm......Page 109
7.1 A simple case of the Metropolis algorithm......Page 110
7.1.1 A politician stumbles upon the Metropolis algorithm......Page 111
7.1.3 General properties of a random walk......Page 113
7.1.5 Why it works......Page 116
7.2 The Metropolis algorithm more generally......Page 120
7.2.2 Terminology: Markov chain Monte Carlo......Page 121
7.3 From the sampled posterior to the three goals......Page 122
7.3.1.1 Highest density intervals from random samples......Page 123
7.3.1.2 Using a sample to estimate an integral......Page 124
7.3.3 Model comparison: Estimation of p(D)......Page 125
7.4 MCMC in BUGS......Page 127
7.4.1 Parameter estimation with BUGS......Page 128
7.4.2 BUGS for prediction......Page 130
7.4.3 BUGS for model comparison......Page 131
7.5 Conclusion......Page 132
7.6.1 R code for a home-grown Metropolis......Page 133
7.7 Exercises......Page 135
8 Inferring Two Binomial Proportions via Gibbs Sampling......Page 139
8.1 Prior, likelihood and posterior for two proportions......Page 141
8.2 The posterior via exact formal analysis......Page 142
8.3 The posterior via grid approximation......Page 145
8.4 The posterior via Markov chain Monte Carlo......Page 146
8.4.1 Metropolis algorithm......Page 147
8.4.2 Gibbs sampling......Page 148
8.4.2.1 Disadvantages of Gibbs sampling......Page 151
8.5 Doing it with BUGS......Page 152
8.5.1 Sampling the prior in BUGS......Page 153
8.6 How different are the underlying biases?......Page 154
8.7 Summary......Page 155
8.8.1 R code for grid approximation (Figures 8.1 and 8.2)......Page 156
8.8.2 R code for Metropolis sampler (Figure 8.3)......Page 158
8.8.3 R code for BUGS sampler (Figure 8.6)......Page 161
8.8.4 R code for plotting a posterior histogram......Page 163
8.9 Exercises......Page 165
9 Bernoulli Likelihood with Hierarchical Prior......Page 169
9.1 A single coin from a single mint......Page 170
9.1.1 Posterior via grid approximation......Page 173
9.2 Multiple coins from a single mint......Page 176
9.2.1 Posterior via grid approximation......Page 178
9.2.2 Posterior via Monte Carlo sampling......Page 181
9.2.2.1 Doing it with BUGS......Page 183
9.2.3 Outliers and shrinkage of individual estimates......Page 187
9.2.4 Case study: Therapeutic touch......Page 189
9.3.1 Independent mints......Page 190
9.3.2 Dependent mints......Page 194
9.3.3 Individual differences and meta-analysis......Page 196
9.5.1 Code for analysis of therapeutic-touch experiment......Page 197
9.5.2 Code for analysis of filtration-condensation experiment......Page 200
9.6 Exercises......Page 203
10.1 Model comparison as hierarchical modeling......Page 207
10.2.1 A simple example......Page 209
10.2.2 A realistic example with “pseudopriors”......Page 211
10.2.3 Some practical advice when using transdimensional MCMC withpseudopriors......Page 216
10.3 Model comparison and nested models......Page 218
10.4.1 Comparing methods for MCMC model comparison......Page 220
10.4.2 Summary and caveats......Page 221
10.5 Exercises......Page 222
11 Null Hypothesis Significance Testing......Page 227
11.1.1 When the experimenter intends to fix N......Page 229
11.1.2 When the experimenter intends to fix z......Page 231
11.1.3 Soul searching......Page 232
11.2 Prior knowledge about the coin......Page 234
11.2.2.1 Priors are overt and should influence......Page 235
11.3.1 NHST confidence interval......Page 236
11.4 Multiple comparisons......Page 239
11.4.1 NHST correction for experimentwise error......Page 240
11.4.2 Just one Bayesian posterior no matter how you look at......Page 242
11.5.1 Planning an experiment......Page 243
11.5.2 Exploring model predictions (posterior predictive check)......Page 244
11.6 Exercises......Page 245
12 Bayesian Approaches to Testing a Point (“Null”) Hypothesis......Page 251
12.1.1 Is a null value of a parameter among the credible values?......Page 252
12.1.2 Is a null value of a difference among the credible values?......Page 253
12.1.2.1 Differences of correlated parameters......Page 254
12.1.3 Region of Practical Equivalence (ROPE)......Page 256
12.2 The model-comparison (two-prior) approach......Page 257
12.2.1 Are the biases of two coins equal or not?......Page 258
12.2.1.1 Formal analytical solution......Page 259
12.2.1.2 Example application......Page 260
12.2.2 Are different groups equal or not?......Page 261
12.3.2 Recommendations......Page 263
12.4.1 R code for Figure 12.5......Page 264
12.5 Exercises......Page 267
13 Goals, Power, and Sample Size......Page 271
13.1.1 Goals and Obstacles......Page 272
13.1.2 Power......Page 273
13.1.3 Sample Size......Page 274
13.2 Sample size for a single coin......Page 276
13.2.1 When the goal is to exclude a null value......Page 277
13.2.2 When the goal is precision......Page 278
13.3 Sample size for multiple mints......Page 279
13.4 Power: prospective, retrospective, and replication......Page 281
13.4.1 Power analysis requires verisimilitude of simulated data......Page 282
13.5 The importance of planning......Page 283
13.6.1 Sample size for a single coin......Page 284
13.6.2 Power and sample size for multiple mints......Page 286
13.7 Exercises......Page 293
III The Generalized Linear Model......Page 301
14 Overview of the Generalized Linear Model......Page 303
14.1.1 Predictor and predicted variables......Page 304
14.1.2 Scale types: metric, ordinal, nominal......Page 305
14.1.3 Linear function of a single metric predictor......Page 306
14.1.4 Additive combination of metric predictors......Page 308
14.1.5 Nonadditive interaction of metric predictors......Page 310
14.1.6.1 Linear model for a single nominal predictor......Page 312
14.1.6.2 Additive combination of nominal predictors......Page 314
14.1.6.3 Nonadditive interaction of nominal predictors......Page 315
14.1.7 Linking combined predictors to the predicted......Page 316
14.1.7.1 The sigmoid (a.k.a. logistic) function......Page 317
14.1.7.2 The cumulative normal (a.k.a. Phi) function......Page 319
14.1.9 Formal expression of the GLM......Page 320
14.2 Cases of the GLM......Page 323
14.2.1 Two or more nominal variables predicting frequency......Page 325
14.3 Exercises......Page 327
15 Metric Predicted Variable on a Single Group......Page 329
15.1.1 Solution by mathematical analysis......Page 330
15.1.2 Approximation by MCMC in BUGS......Page 334
15.1.3 Outliers and robust estimation: The t distribution......Page 335
15.1.4 When the data are non-normal: Transformations......Page 338
15.2 Repeated measures and individual differences......Page 340
15.2.1 Hierarchical model......Page 342
15.2.2 Implementation in BUGS......Page 343
15.4.1 Estimating the mean and precision of a normal likelihood......Page 345
15.4.2 Repeated measures: Normal across and normal within......Page 347
15.5 Exercises......Page 350
16 Metric Predicted Variable with One Metric Predictor......Page 355
16.1 Simple linear regression......Page 356
16.1.1 The hierarchical model and BUGS code......Page 358
16.1.1.1 Standardizing the data for MCMC sampling......Page 359
16.1.1.2 Initializing the chains......Page 360
16.1.2 The posterior: How big is the slope?......Page 361
16.1.3 Posterior prediction......Page 362
16.2 Outliers and robust regression......Page 364
16.3 Simple linear regression with repeated measures......Page 366
16.4 Summary......Page 369
16.5.1 Data generator for height and weight......Page 370
16.5.2 BRugs: Robust linear regression......Page 371
16.5.3 BRugs: Simple linear regression with repeated measures......Page 374
16.6 Exercises......Page 378
17 Metric Predicted Variable with Multiple Metric Predictors......Page 383
17.1.1 The perils of correlated predictors......Page 384
17.1.2 The model and BUGS program......Page 387
17.1.3 The posterior: How big are the slopes?......Page 388
17.1.4 Posterior prediction......Page 390
17.2 Hyperpriors and shrinkage of regression coefficients......Page 391
17.2.1 Informative priors, sparse data, and correlated predictors......Page 393
17.3 Multiplicative interaction of metric predictors......Page 395
17.3.1 The hierarchical model and BUGS code......Page 396
17.3.2 Interpreting the posterior......Page 397
17.4 Which predictors should be included?......Page 400
17.5.1 Multiple linear regression......Page 402
17.5.2 Multiple linear regression with hyperprior on coefficients......Page 406
17.6 Exercises......Page 411
18 Metric Predicted Variable with One Nominal Predictor......Page 413
18.1 Bayesian oneway ANOVA......Page 414
18.1.1 The hierarchical prior......Page 415
18.1.2 Doing it with R and BUGS......Page 416
18.1.3 A worked example......Page 418
18.1.3.1 Contrasts and complex comparisons......Page 419
18.1.3.2 Is there a difference?......Page 420
18.2 Multiple comparisons......Page 421
18.3 Two group Bayesian ANOVA and the NHST t test......Page 424
18.4.1 Bayesian oneway ANOVA......Page 425
18.5 Exercises......Page 429
19 Metric Predicted Variable with Multiple Nominal Predictors......Page 433
19.1 Bayesian multi-factor ANOVA......Page 434
19.1.1 Interaction of nominal predictors......Page 435
19.1.2 The hierarchical prior......Page 436
19.1.3 An example in R and BUGS......Page 437
19.1.4.1 Metric predictors and ANCOVA......Page 440
19.1.4.2 Interaction contrasts......Page 441
19.1.5 Non-crossover interactions, rescaling, and homogeneous variances......Page 442
19.2 Repeated measures, a.k.a. within-subject designs......Page 444
19.2.1 Why use a within-subject design? And why not?......Page 446
19.3.1 Bayesian two-factor ANOVA......Page 447
19.4 Exercises......Page 456
20 Dichotomous Predicted Variable......Page 461
20.1 Logistic regression......Page 462
20.1.2 Doing it in R and BUGS......Page 463
20.1.3 Interpreting the posterior......Page 464
20.1.6 Hyperprior across regression coefficients......Page 466
20.2 Interaction of predictors in logistic regression......Page 467
20.3 Logistic ANOVA......Page 468
20.4 Summary......Page 470
20.5.1 Logistic regression code......Page 471
20.5.2 Logistic ANOVA code......Page 475
20.6 Exercises......Page 480
21 Ordinal Predicted Variable......Page 483
21.1.2 The mapping from metric x to ordinal y......Page 484
21.1.3 The parameters and their priors......Page 486
21.1.5 Posterior prediction......Page 487
21.2 Some examples......Page 488
21.2.1 Why are some thresholds outside the data?......Page 490
21.3 Interaction......Page 492
21.5 R code......Page 493
21.6 Exercises......Page 498
22 Contingency Table Analysis......Page 501
22.1.2 The exponential link function......Page 502
22.1.3 The Poisson likelihood......Page 505
22.1.4 The parameters and the hierarchical prior......Page 506
22.2.1 Credible intervals on cell probabilities......Page 507
22.3 Log linear models for contingency tables......Page 508
22.4 R code for Poisson exponential model......Page 509
22.5 Exercises......Page 516
23 Tools in the Trunk......Page 519
23.1.1 Essential points......Page 520
23.1.3 Helpful points......Page 521
23.2 MCMC burn-in and thinning......Page 522
23.3.2 R code for computing HDI of a MCMC sample......Page 525
23.3.3 R code for computing HDI of a function......Page 527
23.4.1 Examples......Page 528
23.4.2 Reparameterization of two parameters......Page 529
References......Page 531
Index......Page 539