Probability and Statistics

Cover......Page 1
Title Page......Page 4
Copyright Page......Page 5
Contents......Page 8
Preface......Page 12
Acknowledgments......Page 14
1.1 The History of Probability......Page 16
The Frequency Interpretation of Probability......Page 17
The Subjective Interpretation of Probability......Page 18
Types of Experiments......Page 20
The Sample Space......Page 21
Relations of Set Theory......Page 22
Operations of Set Theory......Page 23
Summary......Page 30
Axioms and Basic Theorems......Page 31
Further Properties of Probability......Page 33
Summary......Page 35
Requirements of Probabilities......Page 37
Simple Sample Spaces......Page 38
1.7 Counting Methods......Page 40
Multiplication Rule......Page 41
Permutations......Page 42
The Birthday Problem......Page 45
Summary......Page 46
Combinations......Page 47
Binomial Coefficients......Page 49
Summary......Page 55
1.9 Multinomial Coefficients......Page 57
Summary......Page 60
1.10 The Probability of a Union of Events......Page 61
The Union of a Finite Number of Events......Page 62
Summary......Page 65
Perfect Forecasts......Page 66
Guaranteed Winners......Page 67
1.12 Supplementary Exercises......Page 68
2.1 The Definition of Conditional Probability......Page 70
The Multiplication Rule for Conditional Probabilities......Page 73
Conditional Probability and Partitions......Page 75
Summary......Page 79
Definition of Independence......Page 81
Independence of Two Events......Page 82
Independence of Several Events......Page 83
Conditionally Independent Events......Page 87
Summary......Page 90
2.3 Bayes’ Theorem......Page 91
Statement, Proof, and Examples of Bayes’ Theorem......Page 92
Prior and Posterior Probabilities......Page 95
Summary......Page 99
Statement of the Problem......Page 101
Solution of the Problem......Page 102
Summary......Page 104
2.5 Supplementary Exercises......Page 105
Definition of a Random Variable......Page 108
The Distribution of a Random Variable......Page 109
Discrete Distributions......Page 110
Uniform Distributions on Integers......Page 112
Binomial Distributions......Page 113
Summary......Page 114
The Probability Density Function......Page 115
Nonuniqueness of the p.d.f.......Page 117
Uniform Distributions on Intervals......Page 118
Other Continuous Distributions......Page 119
Summary......Page 121
3.3 The Cumulative Distribution Function......Page 122
Definition and Basic Properties......Page 123
Determining Probabilities from the Distribution Function......Page 125
The c.d.f. of a Continuous Distribution......Page 126
The Quantile Function......Page 127
Summary......Page 131
Discrete Joint Distributions......Page 133
Continuous Joint Distributions......Page 135
Mixed Bivariate Distributions......Page 138
Bivariate Cumulative Distribution Functions......Page 140
Summary......Page 143
Deriving a Marginal p.f. or a Marginal p.d.f.......Page 145
Independent Random Variables......Page 149
Summary......Page 155
3.6 Conditional Distributions......Page 156
Discrete Conditional Distributions......Page 157
Continuous Conditional Distributions......Page 158
Construction of the Joint Distribution......Page 161
Summary......Page 165
3.7 Multivariate Distributions......Page 167
Joint Distributions......Page 168
Mixed Distributions......Page 170
Marginal Distributions......Page 171
Independent Random Variables......Page 173
Conditional Distributions......Page 174
Histograms......Page 179
Summary......Page 181
3.8 Functions of a Random Variable......Page 182
Random Variable with a Continuous Distribution......Page 183
The Probability Integral Transformation......Page 184
Simulation......Page 185
Summary......Page 188
Random Variables with a Discrete Joint Distribution......Page 190
Random Variables with a Continuous Joint Distribution......Page 192
Summary......Page 201
Markov Chains......Page 203
The Transition Matrix......Page 205
The Initial Distribution......Page 211
Stationary Distributions......Page 212
Summary......Page 215
3.11 Supplementary Exercises......Page 217
Expectation for a Discrete Distribution......Page 222
Expectation for a Continuous Distribution......Page 224
Interpretation of the Expectation......Page 225
The Expectation of a Function......Page 227
Summary......Page 231
Basic Theorems......Page 232
Expectation of a Product of Independent Random Variables......Page 237
Summary......Page 239
4.3 Variance......Page 240
Definitions of the Variance and the Standard Deviation......Page 241
Properties of the Variance......Page 243
The Variance of a Binomial Distribution......Page 246
Interquartile Range......Page 247
Summary......Page 248
Existence of Moments......Page 249
Moment Generating Functions......Page 251
Properties of Moment Generating Functions......Page 252
Summary......Page 255
The Median......Page 256
Comparison of the Mean and the Median......Page 257
Minimizing the Mean Squared Error......Page 259
Minimizing the Mean Absolute Error......Page 260
Summary......Page 261
Covariance......Page 263
Correlation......Page 265
Properties of Covariance and Correlation......Page 266
Summary......Page 270
Definition and Basic Properties......Page 271
Prediction......Page 275
Summary......Page 278
Utility Functions......Page 280
Examples of Utility Functions......Page 282
Selling a Lottery Ticket......Page 283
Some Statistical Decision Problems......Page 284
Summary......Page 285
4.9 Supplementary Exercises......Page 287
5.2 The Bernoulli and Binomial Distributions......Page 290
The Bernoulli Distributions......Page 291
The Binomial Distributions......Page 292
Summary......Page 295
Definition and Examples......Page 296
The Mean and Variance for a Hypergeometric Distribution......Page 297
Comparison of Sampling Methods......Page 298
Summary......Page 301
Definition and Properties of the Poisson Distributions......Page 302
The Poisson Approximation to Binomial Distributions......Page 306
Poisson Processes......Page 307
Summary......Page 310
Definition and Interpretation......Page 312
The Geometric Distributions......Page 313
Properties of Negative Binomial and Geometric Distributions......Page 314
Summary......Page 316
Importance of the Normal Distributions......Page 317
Properties of Normal Distributions......Page 318
The Standard Normal Distribution......Page 322
Comparisons of Normal Distributions......Page 323
Linear Combinations of Normally Distributed Variables......Page 324
The Lognormal Distributions......Page 327
Summary......Page 329
The Gamma Function......Page 331
The Gamma Distributions......Page 334
The Exponential Distributions......Page 336
Life Tests......Page 337
Relation to the Poisson Process......Page 338
Summary......Page 340
The Beta Function......Page 342
Definition of the Beta Distributions......Page 343
Moments of Beta Distributions......Page 344
Summary......Page 347
Definition and Derivation of Multinomial Distributions......Page 348
Relation between the Multinomial and Binomial Distributions......Page 350
Means, Variances, and Covariances......Page 351
Definition and Derivation of Bivariate Normal Distributions......Page 352
Properties of Bivariate Normal Distributions......Page 355
Linear Combinations......Page 357
Summary......Page 358
5.11 Supplementary Exercises......Page 360
6.1 Introduction......Page 362
The Markov and Chebyshev Inequalities......Page 363
Properties of the Sample Mean......Page 365
The Law of Large Numbers......Page 367
Summary......Page 373
Statement of the Theorem......Page 375
The Delta Method......Page 379
Summary......Page 384
Approximating a Discrete Distribution by a Continuous Distribution......Page 386
Approximating a Bar Chart......Page 387
Summary......Page 389
6.5 Supplementary Exercises......Page 390
Probability and Statistical Models......Page 391
Examples of Statistical Inference......Page 394
General Classes of Inference Problems......Page 395
Definition of a Statistic......Page 396
References......Page 399
The Prior Distribution......Page 400
The Posterior Distribution......Page 402
The Likelihood Function......Page 405
Sequential Observations and Prediction......Page 406
Summary......Page 408
Sampling from a Bernoulli Distribution......Page 409
Sampling from a Poisson Distribution......Page 412
Sampling from a Normal Distribution......Page 413
Sampling from an Exponential Distribution......Page 416
Improper Prior Distributions......Page 417
Summary......Page 420
Nature of an Estimation Problem......Page 423
Definition of a Bayes Estimator......Page 424
Different Loss Functions......Page 425
The Bayes Estimate for Large Samples......Page 427
More General Parameters and Estimators......Page 429
Summary......Page 430
Introduction......Page 432
Definition of a Maximum Likelihood Estimator......Page 433
Examples of Maximum Likelihood Estimators......Page 434
Summary......Page 440
Invariance......Page 441
Consistency......Page 442
Numerical Computation......Page 443
Method of Moments......Page 445
M.L.E.’s and Bayes Estimators......Page 447
Definition of a Sufficient Statistic......Page 458
The Factorization Criterion......Page 459
Summary......Page 462
Definition of Jointly Sufficient Statistics......Page 464
Minimal Sufficient Statistics......Page 466
Maximum Likelihood Estimators and Bayes Estimators as Sufficient Statistics
......Page 468
Summary......Page 469
The Mean Squared Error of an Estimator......Page 470
Conditional Expectation When a Sufficient Statistic Is Known......Page 471
Summary......Page 475
7.10 Supplementary Exercises......Page 476
Statistics and Estimators......Page 479
Purpose of the Sampling Distribution......Page 480
Summary......Page 483
Definition of the Distributions......Page 484
Properties of the Distributions......Page 485
Summary......Page 487
Independence of the Sample Mean and Sample Variance......Page 488
Estimation of the Mean and Standard Deviation......Page 490
Summary......Page 494
Definition of the Distributions......Page 495
Relation to Random Samples from a Normal Distribution......Page 496
Relation to the Cauchy Distribution and to the Standard Normal Distribution
......Page 497
Summary......Page 499
Confidence Intervals for the Mean of a Normal Distribution......Page 500
One-Sided Confidence Intervals......Page 503
Confidence Intervals for Other Parameters......Page 504
Summary......Page 508
The Precision of a Normal Distribution......Page 510
The Marginal Distribution of the Mean......Page 513
A Numerical Example......Page 515
Improper Prior Distributions......Page 517
Summary......Page 519
Definition of an Unbiased Estimator......Page 521
Unbiased Estimation of the Variance......Page 523
Summary......Page 527
Definition and Properties of Fisher Information......Page 529
The Information Inequality......Page 533
Efficient Estimators......Page 536
Properties of Maximum Likelihood Estimators for Large Samples......Page 537
Summary......Page 542
8.9 Supplementary Exercises......Page 543
The Null and Alternative Hypotheses......Page 545
Simple and Composite Hypotheses......Page 546
The Critical Region and Test Statistics......Page 547
The Power Function and Types of Error......Page 548
Making a Test Have a Specific Significance Level......Page 551
The p-value......Page 553
Equivalence of Tests and Confidence Sets......Page 555
Likelihood Ratio Tests......Page 558
Summary......Page 562
Introduction......Page 565
Optimal Tests......Page 566
Summary......Page 572
Definition of a Uniformly Most Powerful Test......Page 574
Monotone Likelihood Ratio......Page 575
One-Sided Alternatives......Page 577
Two-Sided Alternatives......Page 580
9.4 Two-Sided Alternatives......Page 582
General Form of the Procedure......Page 583
Selection of the Test Procedure......Page 584
Other Distributions......Page 585
Composite Null Hypothesis......Page 586
Summary......Page 589
Testing Hypotheses about the Mean of a Normal Distribution When the Variance Is Unknown......Page 591
Properties of the t Tests......Page 592
The Paired t Test......Page 595
Testing with a Two-Sided Alternative......Page 596
Summary......Page 600
The Two-Sample t Test......Page 602
Power of the Test......Page 605
Two-Sided Alternatives......Page 606
Summary......Page 610
Definition of the F Distribution......Page 612
Properties of the F Distributions......Page 613
Properties of F Tests......Page 614
Two-Sided Alternative......Page 616
Summary......Page 619
Simple Null and Alternative Hypotheses......Page 620
Tests Based on the Posterior Distribution......Page 622
One-Sided Hypotheses......Page 623
Two-Sided Alternatives......Page 625
Testing the Mean of a Normal Distribution with Unknown Variance......Page 626
Comparing the Means of Two Normal Distributions......Page 627
Comparing the Variances of Two Normal Distributions......Page 629
Summary......Page 630
The Relationship between Level of Significance and Sample Size......Page 632
Statistically Significant Results......Page 634
Summary......Page 635
9.10 Supplementary Exercises......Page 636
Description of Nonparametric Problems......Page 639
Categorical Data......Page 640
The χ2 Test......Page 641
Testing Hypotheses about a Continuous Distribution......Page 643
Summary......Page 646
Composite Null Hypotheses......Page 648
The χ2 Test for Composite Null Hypotheses......Page 650
Testing Whether a Distribution Is Normal......Page 651
Summary......Page 654
Independence in Contingency Tables......Page 656
The χ2 Test of Independence......Page 658
Summary......Page 660
Samples from Several Populations......Page 662
The χ2 Test of Homogeneity......Page 663
Comparing Two or More Proportions......Page 664
Correlated 2 × 2 Tables......Page 665
Summary......Page 667
An Example of the Paradox......Page 668
The Paradox Explained......Page 670
Summary......Page 671
The Sample Distribution Function......Page 672
The Kolmogorov-Smirnov Test of a Simple Hypothesis......Page 675
The Kolmogorov-Smirnov Test for Two Samples......Page 678
10.7 Robust Estimation......Page 681
Estimating the Median......Page 682
Contaminated Normal Distributions......Page 683
Trimmed Means......Page 684
Robust Estimation of Scale......Page 685
M-Estimators of the Median......Page 686
Comparison of the Estimators......Page 689
Large-Sample Properties of Sample Quantiles......Page 691
Summary......Page 692
One-Sample Procedures......Page 693
Comparing Two Distributions......Page 695
Ties......Page 697
Summary......Page 699
10.9 Supplementary Exercises......Page 701
Fitting a Straight Line......Page 704
The Least-Squares Line......Page 706
Fitting a Polynomial by the Method of Least Squares......Page 708
Fitting a Linear Function of Several Variables......Page 710
Summary......Page 712
Regression Functions......Page 713
Simple Linear Regression......Page 715
The Distribution of the Least-Squares Estimators......Page 716
Prediction......Page 718
Summary......Page 721
Joint Distribution of the Estimators......Page 722
Tests of Hypotheses about the Regression Coefficients......Page 726
Confidence Intervals......Page 730
The Analysis of Residuals......Page 732
Summary......Page 741
Improper Priors for Regression Parameters......Page 744
Prediction Intervals......Page 747
Tests of Hypotheses......Page 748
Summary......Page 750
The General Linear Model......Page 751
Maximum Likelihood Estimators......Page 753
Explicit Form of the Estimators......Page 754
Mean Vector and Covariance Matrix......Page 756
Testing Hypotheses......Page 760
Prediction......Page 762
Multiple R²
......Page 763
Analysis of Residuals......Page 764
Summary......Page 767
The One-Way Layout......Page 769
Partitioning a Sum of Squares......Page 772
Testing Hypotheses......Page 774
Analysis of Residuals......Page 775
Summary......Page 776
The Two-Way Layout with One Observation in Each Cell......Page 778
Estimating the Parameters......Page 780
Partitioning the Sum of Squares......Page 781
Testing Hypotheses......Page 783
Summary......Page 785
The Two-Way Layout with K Observations in Each Cell......Page 787
Partitioning the Sum of Squares......Page 790
Testing Hypotheses......Page 791
The Two-Way Layout with Unequal Numbers of Observationsin the Cells......Page 795
Summary......Page 796
11.9 Supplementary Exercises......Page 798
Proof of Concept......Page 802
Examples in which Simulation Might Help......Page 803
Summary......Page 805
Examples of Simulation......Page 806
Which Mean Do You Mean?......Page 809
Assessing Uncertainty about Simulation Results......Page 811
Summary......Page 817
The Probability Integral Transformation......Page 819
Acceptance/Rejection......Page 820
Some Examples Involving Simulation of Common Distributions......Page 823
Simulating a Discrete Random Variable......Page 827
Summary......Page 830
12.4 Importance Sampling......Page 831
Summary......Page 836
12.5 Markov Chain Monte Carlo......Page 838
The Gibbs Sampling Algorithm......Page 839
When Does the Markov Chain Converge?......Page 840
Some Examples......Page 842
Prediction......Page 848
Summary......Page 851
Introduction......Page 854
The Bootstrap in General......Page 856
The Nonparametric Bootstrap......Page 858
The Parametric Bootstrap......Page 860
Summary......Page 863
12.7 Supplementary Exercises......Page 865
Tables......Page 868
Answers to Odd-Numbered Exercises......Page 880
References......Page 894
C......Page 900
D......Page 901
G......Page 902
K......Page 903
M......Page 904
P......Page 905
S......Page 906
V......Page 907
Z......Page 908