Statistics and Analysis of Scientific Data

Foreword
Preface
Acknowledgments
Contents
Part I Probability, Random Variables and Statistics
1 Theory of Probability
	1.1 Experiments and Events
	1.2 Probability of Events
		1.2.1 The Kolmogorov Axioms
		1.2.2 Frequentist or Classical Method
		1.2.3 Bayesian or Empirical Method
		1.2.4 Fundamental Properties of Probability
	1.3 The Conditional Probability
	1.4 Statistical Independence
	1.5 A Classic Experiment: Mendel's Experiments on Plant Hybridization
	1.6 The Total Probability Theorem and Bayes' Theorem
2 Random Variables and Their Distributions
	2.1 Random Variables
	2.2 Probability Distribution Functions
	2.3 Expectations and Moments of a Distribution Function
		2.3.1 The Mean and the Sample Mean
		2.3.2 The Law of Large Numbers
		2.3.3 The Variance and the Sample Variance
	2.4 A Classic Experiment: J.J. Thomson's Discovery of the Electron
	2.5 Covariance and Correlation Between Random Variables
		2.5.1 Joint Distribution and Moments of Two Random Variables
		2.5.2 Statistical Independence of Random Variables
	2.6 The Expectation of the Sample Variance and Sample Covariance
	2.7 A Classic Experiment: Pearson's Collection of Data on Biometric Characteristics
3 Three Fundamental Distributions: Binomial, Gaussian, and Poisson
	3.1 The Binomial Distribution
		3.1.1 Derivation of the Binomial Distribution
		3.1.2 Moments of the Binomial Distribution
	3.2 The Gaussian Distribution
		3.2.1 Derivation of the Gaussian Distribution from the Binomial Distribution
		3.2.2 Moments and Properties of the Gaussian Distribution
	3.3 The Poisson Distribution
		3.3.1 Derivation of the Poisson Distribution
		3.3.2 Moments and Properties of the Poisson Distribution
		3.3.3 The Poisson Distribution and the Poisson Process
	3.4 Comparison of the Binomial, Gaussian, and Poisson Distributions
4 The Distribution of Functions of Random Variables
	4.1 Functions of Random Variables
	4.2 Linear Combination of Random Variables
		4.2.1 Mean and Variance Formulas
		4.2.2 Independent Measurements and the 1/sqrtN Factor
	4.3 The Moment Generating Function
		4.3.1 Properties of the Moment Generating Function
		4.3.2 Moment-Generating Functions of Selected Distributions
	4.4 The Central Limit Theorem
		4.4.1 The Distribution of the Sample Mean of Gaussian Measurements
		4.4.2 The Distribution of the Sum of Standard Uniform Random Variables
		4.4.3 Certain Limitations of the Central Limit Theorem
	4.5 The Distribution of Functions of Random Variables
		4.5.1 The Method of Change of Variables
		4.5.2 Direct Method Using the Distribution Function
5 Error Propagation and Simulation  of Random Variables
	5.1 The Mean of Functions of Random Variables
	5.2 The Variance of Functions of Random Variables and Error Propagation Formulas
		5.2.1 Sum and Product of a Constant
		5.2.2 Weighted Sum of Two Variables
		5.2.3 Product and Division of Two Random Variables
		5.2.4 Power of a Random Variable
		5.2.5 Exponential of a Random Variable
		5.2.6 Logarithm of a Random Variable
	5.3 The Quantile Function and Simulation of Random Variables
		5.3.1 General Method to Simulate a Variable
		5.3.2 Simulation of a Gaussian Variable
6 Maximum Likelihood and Other Methods to Estimate Variables
	6.1 Estimating Random Variables with Data
	6.2 The Maximum-Likelihood Method
		6.2.1 Maximum-Likelihood Methods for a Gaussian Variable
		6.2.2 Maximum-Likelihood Estimate of the Gaussian Mean for Non-uniform Uncertainties
	6.3 The Maximum-Likelihood Method for the Poisson and Other Distributions
	6.4 Method of Moments
	6.5 Method of Maximum Entropy
7 Methods of Inference and Confidence Intervals of Random Variables
	7.1 Quantiles and Confidence Intervals
	7.2 Fiducial Inference
	7.3 Confidence Intervals for a Gaussian Variable
	7.4 Upper and Lower Limits for a Gaussian Variable
	7.5 Confidence Intervals for the Mean of a Poisson Variable
	7.6 The Gehrels Approximation for Poisson Upper and Lower Limits
	7.7 Bayesian Methods of Inference
		7.7.1 Bayesian Expectation of the Poisson Mean
		7.7.2 Bayesian Confidence Intervals for a Poisson Variable
8 Average Values of Random Variables
	8.1 Point Estimates and Average Values
	8.2 Linear and Weighted Averages
	8.3 The Median
	8.4 The Logarithmic Average and Fractional Errors
		8.4.1 The Log-Normal Distribution
		8.4.2 The Weighted Logarithmic Average
		8.4.3 The Relative-Error Weighted Average
	Problems
Part II Hypothesis Testing, Regression and Parameter Estimation
9 Hypothesis Testing and Fundamental Statistics
	9.1 Statistics and Hypothesis Testing
	9.2 The P-Value of a Statistical Analysis
	9.3 The χ2 Statistic
		9.3.1 The Probability Distribution Function
		9.3.2 Moments and Other Properties
		9.3.3 Hypothesis Testing
	9.4 The Distribution of the Sample Variance
	9.5 The F-Statistic
		9.5.1 The Probability Distribution Function
		9.5.2 Moments and Other Properties
		9.5.3 Hypothesis Testing
	9.6 The Sampling Distribution of the Mean and Student's t-Statistic
		9.6.1 Student's t-Statistic for the Sample Mean
		9.6.2 Hypothesis Testing with the t-Statistic
		9.6.3 Comparison of Two Sample Means and Hypothesis Testing
10 Contingency Tables and Diagnostic Tests
	10.1 A Classic Experiment: The 1915 Greenwood and Yule Inoculation Statistics
	10.2 2 times2 Contingency Tables
		10.2.1 The χ2 Test
		10.2.2 χ2 Test with the Yates Continuity Correction
		10.2.3 The Fisher Exact Test for  2 times2 Contingency Tables
		10.2.4 Exact Tests Based on the Binomial Distribution
	10.3 Higher Dimension r timesc Contingency Tables
	10.4 Binary Diagnostic Tests
		10.4.1 Sensitivity, Specificity, and Likelihood Ratios
		10.4.2 Posterior Probabilities: The Positive and Negative Predictive Values
		10.4.3 Change in Posterior Probability with Repeated Testing
	10.5 Vaccine Efficacy
11 Linear and Non-linear Regression  for Gaussian Data
	11.1 Measurement of Pairs of Variables and Regression
	11.2 Regression Using Maximum Likelihood for Gaussian Data
	11.3 Linear Regression with Gaussian Data
	11.4 Multiple Linear Regression
	11.5 Linear Regression with Uniform Variance
		11.5.1 Alternative form of the Solution with Sample Moments
		11.5.2 Choice of Independent Variable
	11.6 A Classic Experiment: Edwin Hubble's Discovery of the Expansion of the Universe
	11.7 Non-linear Regression
12 Goodness of Fit and Parameter Uncertainty for Gaussian Data
	12.1 The χ2min Goodness-of-Fit Statistic
	12.2 Data with No Errors and the Model Sample Variance
	12.3 The Δχ2 Statistic
	12.4 Confidence Intervals of Model Parameters
	12.5 Confidence Intervals on a Reduced Number  of Parameters
13 Multi-variable Regression
	13.1 Multi-variable Datasets
	13.2 A Classic Experiment: The R.A. Fisher and E. Anderson Measurements of Iris Characteristics
	13.3 The Multi-variable Linear Regression
	13.4 Multi-variable Linear Regression with Uniform Variance
	13.5 Goodness of Fit of Multi-variable Regression
	13.6 Tests for the Significance of Multiple Regression Coefficients
		13.6.1 t-Test for the Significance of Model Components
		13.6.2 F-Test for the Significance of the a1, …, am Parameters
		13.6.3 The Coefficient of Determination
14 The Linear Correlation Coefficient
	14.1 Linear Regression and Choice of the Independent Variable
	14.2 The Linear Correlation Coefficient
	14.3 Sampling Distribution of r and Hypothesis Testing
	14.4 Distribution of the Coefficient of Determination R2 and of r2
15 Low-Count Poisson Data and the Cash Statistic
	15.1 Poisson Data with Integer-Valued Variables
	15.2 Likelihood of Poisson Data and the Cash Statistic
	15.3 Distribution of the Cash Statistic for a Fully Specified Model
		15.3.1 Asymptotic Values for the Mean and Variance
		15.3.2 Analytical Approximations for the Mean and Variance
		15.3.3 Other Useful Formulas for the Moments
	15.4 Hypothesis Testing with the C Statistic
16 Maximum Likelihood Methods and Parameter Estimation with the Cash Statistic
	16.1 Maximum Likelihood Methods for Poisson Data
	16.2 Linear Regression with Poisson Data
		16.2.1 The Standard Linear Model
		16.2.2 A Factorized Linear Model with a Semi-Analytical Solution
		16.2.3 An Extended Linear Model
		16.2.4 Non-Uniform Bin Size and Gaps in the Data
	16.3 Goodness of Fit and Hypothesis Testing with the Cash Statistic
		16.3.1 The Wilks Theorem on the Likelihood Ratio
		16.3.2 The Large-Count Regime
		16.3.3 The Low-Count Regime
		16.3.4 Approximate Methods in the Low-Count Regimes
	16.4 Parameter Estimation with the C statistic
	16.5 Biases Using χ2 for Poisson Data in the Large-Count Limit
17 Systematic Errors and Intrinsic Scatter
	17.1 What to Do When the Goodness-of-Fit Test Fails
	17.2 Intrinsic Scatter and the Debiased Variance
		17.2.1 Direct Calculation of the Intrinsic Scatter
		17.2.2 Alternative Method for Gaussian Data
	17.3 Systematic Errors
	17.4 Estimate of Model Parameters with Systematic Errors or Intrinsic Scatter
18 Regression with Bivariate Errors
	18.1 Two-Variable Data with Bivariate Errors
	18.2 Least-Squares Linear Fit to Data with Bivariate Errors
	18.3 Linear Fit Using Bivariate Errors in the χ2 Statistic
19 Model and Data Comparison
	19.1 The χ2min Statistic and the F-Test for Gaussian Data
	19.2 F-Test for Two Independent χ2 Measurements
	19.3 F-Test for an Additional Model Component
	19.4 Kolmogorov–Smirnov Tests
		19.4.1 Comparison of Data to a Model
		19.4.2 Two-Sample Kolmogorov–Smirnov Test
Part III Monte Carlo Methods
20 Monte Carlo and Re-sampling Methods
	20.1 What is a Monte Carlo Analysis?
	20.2 Traditional Monte Carlo Integration
	20.3 Hit-or-Miss Monte Carlo Methods
	20.4 Simulation of Random Variables
	20.5 Re-sampling Methods
	20.6 The Jackknife Method
	20.7 The Bootstrap Method
21 Introduction to Markov Chains
	21.1 Stochastic Processes and Markov Chains
	21.2 Mathematical Properties of Markov Chains
	21.3 Recurrent and Transient States
	21.4 Limiting Probabilities and Stationary Distribution
	21.5 Ergodic Averages and Variance Estimates
22 Markov Chain Monte Carlo
	22.1 Introduction to Markov Chain Monte Carlo Methods
	22.2 Markov Chain Monte Carlo for Regression Analysis
	22.3 The Metropolis–Hastings MCMC
	22.4 The Gibbs Sampler
	22.5 Convergence of Markov Chain Monte Carlo
	22.6 The Geweke z-Score Convergence Test
	22.7 The Gelman–Rubin Convergence Test
	22.8 The Raftery–Lewis Diagnostic
	22.9 Inference with MCMC
23 Numerical Methods and python Codes
	23.1 Analytical and Numerical Methods
	23.2 Introduction to python
	23.3 General Features of python Codes for this Textbook
		23.3.1 Structure of the Codes
		23.3.2 Functions, Library Import and Settings
		23.3.3 Data Associated with the Codes
	23.4 Description of Codes
	23.5 Numerical Methods for Tables in Appendix
Appendix  Appendix: Numerical Tables
A.1  The Gaussian Distribution and the Error Function
A.2  Upper and Lower Limits for a Poisson Distribution
A.3  The Gamma and Beta Distributions and Functions
A.4  The χ2 Distribution
A.5  The F Distribution
A.6  The Student t Distribution
A.7  The Linear Correlation Coefficient r
A.8  The Kolmogorov–Smirnov Statistics
Appendix  References
Index