Bayesian Logical Data Analysis For The Physical Sciences - A Comparative Approach With Mathematica

Cover......Page 1
Abstract......Page 3
Title page......Page 5
Contents......Page 7
Preface......Page 15
Acknowledgements......Page 19
1.1 Scientific inference......Page 21
1.2 Inference requires a probability theory......Page 22
1.2.1 The two rules for manipulating probabilities......Page 24
1.3.1 Discrete hypothesis space......Page 25
1.3.2 Continuous hypothesis space......Page 26
1.3.3 Bayes' theorem - model of the learning process......Page 27
1.3.4 Example of the use of Bayes' theorem......Page 28
1.4 Probability and frequency......Page 30
1.4.1 Example: incorporating frequency information......Page 31
1.5 Marginalization......Page 32
1.6 The two basic problems in statistical inference......Page 35
1.7 Advantages of the Bayesian approach......Page 36
1.8 Problems......Page 37
2.2.1 Logical propositions......Page 41
2.2.3 Truth tables and Boolean algebra......Page 42
2.2.4 Deductive inference......Page 44
2.3 Brief history......Page 45
2.4 An adequate set of operations......Page 46
2.4.1 Examination of a logic function......Page 47
2.5 Operations for plausible inference......Page 49
2.5.2 Development of the product rule......Page 50
2.5.3 Development of sum rule......Page 54
2.5.4 Qualitative properties of product and sum rules......Page 56
2.6 Uniqueness of the product and sum rules......Page 57
2.8 Problems......Page 59
3.2 Basics......Page 61
3.3 Parameter estimation......Page 63
3.5 Model comparison and Occam's razor......Page 65
3.6.1 Background information......Page 70
3.7 Odds ratio......Page 72
3.7.1 Choice of prior p(T|M₁,I)......Page 73
3.7.2 Calculation of p(D|M₁,T,I)......Page 75
3.7.5 Odds, Jeffreys prior......Page 78
3.8.1 Sensitivity of odds to T_max......Page 79
3.9 Lessons......Page 81
3.10 Ignorance priors......Page 83
3.11 Systematic errors......Page 85
3.11.1 Systematic error example......Page 86
3.12 Problems......Page 89
4.2 Binomial distribution......Page 92
4.2.2 The gambler's coin problem......Page 95
4.2.3 Bayesian analysis of an opinion poll......Page 97
4.3 Multinomial distribution......Page 99
4.4 Can you really answer that question?......Page 100
4.5 Logical versus causal connections......Page 102
4.6 Exchangeable distributions......Page 103
4.7 Poisson distribution......Page 105
4.7.1 Bayesian and frequentist comparison......Page 107
4.8 Constructing likelihood functions......Page 109
4.8.1 Deterministic model......Page 110
4.8.2 Probabilistic model......Page 111
4.9 Summary......Page 113
4.10 Problems......Page 114
5.2 The concept of a random variable......Page 116
5.3 Sampling theory......Page 117
5.4 Probability distributions......Page 118
5.5 Descriptive properties of distributions......Page 120
5.5.1 Relative line shape measures for distributions......Page 121
5.5.2 Standard random variable......Page 122
5.5.3 Other measures of central tendency and dispersion......Page 123
5.5.4 Median baseline subtraction......Page 124
5.6 Moment generating functions......Page 125
5.7.1 Binomial distribution......Page 127
5.7.2 The Poisson distribution......Page 129
5.7.3 Negative binomial distribution......Page 132
5.8.1 Normal distribution......Page 133
5.8.3 Gamma distribution......Page 136
5.8.3 Beta distribution......Page 137
5.8.5 Negative exponential distribution......Page 138
5.9 Central Limit Theorem......Page 139
5.10 Bayesian demonstration of the Central Limit Theorem......Page 140
5.11 Distribution of the sample mean......Page 144
5.12 Transformation of a random variable......Page 145
5.13 Random and pseudo-random numbers......Page 147
5.13.1 Pseudo-random number generators......Page 151
5.13.2 Tests for randomness......Page 152
5.14 Summary......Page 156
5.15 Problems......Page 157
6.1 Introduction......Page 159
6.2 The χ² distribution......Page 161
6.3 Sample variance S²......Page 163
6.4 The Student's t distribution......Page 167
6.5 F distribution (F-test)......Page 170
6.6.1 Variance σ² known......Page 172
6.6.2 Confidence intervals for μ, unknown variance......Page 176
6.6.3 Confidence intervals: difference of two means......Page 178
6.6.5 Confidence intervals: ratio of two variances......Page 179
6.7 Summary......Page 180
6.8 Problems......Page 181
7.2 Basic idea......Page 182
7.2.1 Hypothesis testing with the χ² statistic......Page 183
7.2.2 Hypothesis test on the difference of two means......Page 187
7.2.3 One-sided and two-sided hypothesis tests......Page 190
7.3 Are two distributions the same?......Page 192
7.3.1 Pearson χ² goodness-of-fit test......Page 193
7.4 Problem with frequentist hypothesis testing......Page 197
7.4.1 Bayesian resolution to optional stopping problem......Page 199
7.5 Problems......Page 201
8.1 Overview......Page 204
8.2 The maximum entropy principle......Page 205
8.3 Shannon's theorem......Page 206
8.4 Alternative justification of MaxEnt......Page 207
8.5.1 Incorporating a prior......Page 210
8.6.1 Lagrange multipliers of variational calculus......Page 211
8.7.1 General properties......Page 212
8.7.2 Uniform distribution......Page 214
8.7.3 Exponential distribution......Page 215
8.7.4 Normal and truncated Gaussian distributions......Page 217
8.7.5 Multivariate Gaussian distribution......Page 222
8.8.1 The kangaroo justification......Page 223
8.8.2 MaxEnt for uncertain constraints......Page 226
8.9 Pixon multiresolution image reconstruction......Page 228
8.10 Problems......Page 231
9.2 Bayesian estimate of a mean......Page 232
9.2.1 Mean: known noise σ......Page 233
9.2.2 Mean: known noise, unequal σ......Page 237
9.2.3 Mean: unknown noise σ......Page 238
9.2.4 Bayesian estimate of σ......Page 244
9.3 Is the signal variable?......Page 247
9.4 Comparison of two independent samples......Page 248
9.4.1 Do the samples differ?......Page 250
9.4.3 Results......Page 253
9.4.4 The difference in means......Page 256
9.4.5 Ratio of the standard deviations......Page 257
9.4.6 Effect of the prior ranges......Page 259
9.5 Summary......Page 260
9.6 Problems......Page 261
10.1 Overview......Page 263
10.2 Parameter estimation......Page 264
10.2.1 Most probable amplitudes......Page 269
10.2.2 More powerful matrix formulation......Page 273
10.3 Regression analysis......Page 276
10.4 The posterior is a Gaussian......Page 277
10.4.1 Joint credible regions......Page 280
10.5.1 Marginalization and the covariance matrix......Page 284
10.5.2 Correlation coefficient......Page 288
10.5.3 More on model parameter errors......Page 292
10.6 Correlated data errors......Page 293
10.7 Model comparison with Gaussian posteriors......Page 295
10.8 Frequentist testing and errors......Page 299
10.8.1 Other model comparison methods......Page 301
10.9 Summary......Page 303
10.10 Problems......Page 304
11.1 Introduction......Page 307
11.2 Asymptotic normal approximation......Page 308
11.3.1 Bayes factor......Page 311
11.3.2 Marginal parameter posteriors......Page 313
11.4 Finding the most probable parameters......Page 314
11.4.1 Simulated annealing......Page 316
11.4.2 Genetic algorithm......Page 317
11.5 Iterative linearization......Page 318
11.5.1 Levenberg-Marquardt method......Page 320
11.5.2 Marquardt's recipe......Page 321
11.6 Mathematica example......Page 322
11.6.1 Model comparison......Page 324
11.6.2 Marginal and projected distributions......Page 326
11.7 Errors in both coordinates......Page 327
11.9 Problems......Page 329
12.1 Overview......Page 332
12.2 Metropolis-Hastings algorithm......Page 333
12.3 Why does Metropolis-Hastings work?......Page 339
12.5 Parallel tempering......Page 341
12.6 Example......Page 342
12.7 Model comparison......Page 346
12.8 Towards an automated MCMC......Page 350
12.9 Extrasolar planet example......Page 351
12.9.1 Model probabilities......Page 355
12.9.2 Results......Page 357
12.10 MCMC robust summary statistic......Page 362
12.11 Summary......Page 366
12.12 Problems......Page 369
13.2 New insights on the periodogram......Page 372
13.2.1 How to compute p(f|D,I)......Page 376
13.3 Strong prior signal model......Page 378
13.4 No specific prior signal model......Page 380
13.4.1 X-ray astronomy example......Page 382
13.4.2 Radio astronomy example......Page 383
13.5 Generalized Lomb-Scargle periodogram......Page 385
13.5.2 Example......Page 387
13.6 Non-uniform sampling......Page 390
13.7 Problems......Page 393
14.1 Overview......Page 396
14.2 Infer a Poisson rate......Page 397
14.2.1 Summary of posterior......Page 398
14.3 Signal + known background......Page 399
14.4 Analysis of ON/OFF measurements......Page 400
14.4.1 Estimating the source rate......Page 401
14.4.2 Source detection question......Page 404
14.5 Time-varying Poisson rate......Page 406
14.6 Problems......Page 408
Appendix A Singular value decomposition......Page 409
B.2 Orthogonal and orthonormal functions......Page 412
B.3 Fourier series and integral transform......Page 414
B.3.1 Fourier series......Page 415
B.3.2 Fourier transform......Page 416
B.4 Convolution and correlation......Page 418
B.4.1 Convolution theorem......Page 419
B.4.2 Correlation theorem......Page 420
B.4.3 Importance of convolution in science......Page 421
B.5 Waveform sampling......Page 423
B.6 Nyquist sampling theorem......Page 424
B.6.1 Astronomy example......Page 426
B.7.1 Graphical development......Page 427
B.7.2 Mathematical development of the DFT......Page 429
B.7.3 Inverse DFT......Page 430
B.8.1 DFT as an approximate Fourier transform......Page 431
B.8.2 Inverse discrete Fourier transform......Page 433
B.9 The Fast Fourier Transform......Page 435
B.10 Discrete convolution and correlation......Page 437
B.10.1 Deconvolving a noisy signal......Page 438
B.10.2 Deconvolution with an optimal Weiner filter......Page 440
B.10.3 Treatment of end effects by zero padding......Page 441
B.11 Accurate amplitudes by zero padding......Page 442
B.12.1 Parseval's theorem and power spectral density......Page 444
B.12.2 Periodogram power-spectrum estimation......Page 445
B.12.3 Correlation spectrum estimation......Page 446
B.13.1 Discrete form of Parseval's theorem......Page 448
B.13.3 Variance of periodogram estimate......Page 449
B.13.5 Reduction of periodogram variance......Page 451
B.14 Problems......Page 452
C.2.1 Evaluation of p(C,S|D₁,D₂,I)......Page 454
C.2.2 Evaluation of p(C,S̄|D₁,D₂,I)......Page 456
C.2.3 Evaluation of p(C̄,S|D₁,D₂,I)......Page 458
C.3 The difference in the means......Page 459
C.3.1 The two-sample problem......Page 460
C.3.2 The Behrens-Fisher problem......Page 461
C.4.1 Estimating the ratio, given the means are the same......Page 462
C.4.2 Estimating the ratio, given the means are different......Page 463
D.1 Derivation of p(s|N_on,I)......Page 465
D.1.1 Evaluation of Num......Page 466
D.1.2 Evaluation of Den......Page 467
D.2 Derivation of the Bayes factor B_{s+b,b}......Page 468
Appendix E Multivariate Gaussian from maximum entropy......Page 470
References......Page 475
Index......Page 481