Advanced statistics with applications in R

Why I Wrote This Book1 Discrete random variables 11.1 Motivating example 11.2 Bernoulli random variable 21.3 General discrete random variable 41.4 Mean and variance 61.4.1 Mechanical interpretation of the mean 71.4.2 Variance 121.5 R basics 151.5.1 Scripts/functions 161.5.2 Text editing in R 171.5.3 Saving your R code 181.5.4 for loop 181.5.5 Vectorized computations 191.5.6 Graphics 231.5.7 Coding and help in R 251.6 Binomial distribution 261.7 Poisson distribution 321.8 Random number generation using sample 381.8.1 Generation of a discrete random variable 381.8.2 Random Sudoku 392 Continuous random variables 432.1 Distribution and density functions 432.1.1 Cumulative distribution function 432.1.2 Empirical cdf 452.1.3 Density function 462.2 Mean, variance, and other moments 482.2.1 Quantiles, quartiles, and the median 542.2.2 The tight confidence range 552.3 Uniform distribution 592.4 Exponential distribution 632.4.1 Laplace or double-exponential distribution 672.4.2 R functions 672.5 Moment generating function 692.5.1 Fourier transform and characteristic function 722.6 Gamma distribution 752.6.1 Relationship to Poisson distribution 772.6.2 Computing the gamma distribution in R 792.6.3 The tight confidence range 792.7 Normal distribution 822.8 Chebyshev's inequality 912.9 The law of large numbers 932.9.1 Four types of stochastic convergence 942.9.2 Integral approximation using simulations 992.10 The central limit theorem 1042.10.1 Why the normal distribution is the most natural symmetric distribution 1122.10.2 CLT on the relative scale 1132.11 Lognormal distribution 1162.11.1 Computation of the tight confidence range 1182.12 Transformations and the delta method 1202.12.1 The delta method 1242.13 Random number generation 1262.13.1 Cauchy distribution 1302.14 Beta distribution 1322.15 Entropy 1342.16 Benford's law: the distribution of the first digit 1382.16.1 Distributions that almost obey Benford's law 1422.17 The Pearson family of distributions 1452.18 Major univariate continuous distributions 1473 Multivariate random variables 1493.1 Joint cdf and density 1493.1.1 Expectation 1543.1.2 Bivariate discrete distribution 1543.2 Independence 1563.2.1 Convolution 1593.3 Conditional density 1683.3.1 Conditional mean and variance 1713.3.2 Mixture distribution and Bayesian statistics 1793.3.3 Random sum 1823.3.4 Cancer tumors grow exponentially 1843.4 Correlation and linear regression 1893.5 Bivariate normal distribution 1983.5.1 Regression as conditional mean 2063.5.2 Variance decomposition and coefficient of determination 2083.5.3 Generation of dependent normal observations 2093.5.4 Copula 2143.6 Joint density upon transformation 2183.7 Geometric probability 2233.7.1 Meeting problem 2243.7.2 Random objects on the square 2253.8 Optimal portfolio allocation 2303.8.1 Stocks do not correlate 2313.8.2 Correlated stocks 2323.8.3 Markowitz bullet 2333.8.4 Probability bullet 2343.9 Distribution of order statistics 2363.10 Multidimensional random vectors 2393.10.1 Multivariate conditional distribution 2453.10.2 Multivariate MGF 2473.10.3 Multivariate delta method 2483.10.4 Multinomial distribution 2514 Four important distributions in statistics 2554.1 Multivariate normal distribution 2554.1.1 Generation of multivariate normal variables 2594.1.2 Conditional distribution 2614.1.3 Multivariate CLT 2684.2 Chi-square distribution 2704.2.1 Noncentral chi-square distribution 2764.2.2 Expectations and variances of quadratic forms 2774.2.3 Kronecker product and covariance matrix 2774.3 t-distribution 2804.3.1 Noncentral t-distribution 2844.4 F-distribution 2865 Preliminary data analysis and visualization 2915.1 Comparison of random variables using the cdf 2915.1.1 ROC curve 2945.1.2 Survival probability 3055.2 Histogram 3125.3 Q-Q plot 3155.3.1 The q-q confidence bands 3195.4 Box plot 3245.5 Kernel density estimation 3255.5.1 Density movie 3315.5.2 3D scatterplots 3335.6 Bivariate normal kernel density 3355.6.1 Bivariate kernel smoother for images 3395.6.2 Smoothed scatterplot 3415.6.3 Spatial statistics for disease mapping 3426 Parameter estimation 3476.1 Statistics as inverse probability 3496.2 Method of moments 3506.2.1 Generalized method of moments 3536.3 Method of quantiles 3576.4 Statistical properties of an estimator 3586.4.1 Unbiasedness 3596.4.2 Mean Square Error 3656.4.3 Multidimensional MSE 3716.4.4 Consistency of estimators 3736.5 Linear estimation 3786.5.1 Estimation of the mean using linear estimator 3796.5.2 Vector representation 3836.6 Estimation of variance and correlation coefficient 3856.6.1 Quadratic estimation of the variance 3866.6.2 Estimation of the covariance and correlation coefficient 3896.7 Least squares for simple linear regression 3986.7.1 Gauss-Markov theorem 4026.7.2 Statistical properties of the OLS estimator under the normal assumption 4046.7.3 The lm function and prediction by linear regression 4066.7.4 Misinterpretation of the coefficient of determination 4106.8 Sufficient statistics and the exponential family of distributions 4156.8.1 Uniformly minimum-variance unbiased estimator 4196.8.2 Exponential family of distributions 4226.9 Fisher information and the Cramer-Rao bound 4336.9.1 One parameter 4346.9.2 Multiple parameters 4406.10 Maximum likelihood 4536.10.1 Basic definitions and examples 4536.10.2 Circular statistics and the von Mises distribution 4716.10.3 Maximum likelihood, sufficient statistics and the exponential family 4756.10.4 Asymptotic properties of ML 4776.10.5 When maximum likelihood breaks down 4856.10.6 Algorithms for log-likelihood function maximization 4986.11 Estimating equations and the M-estimator 5106.11.1 Robust statistics 5167 Hypothesis testing and confidence intervals 5237.1 Fundamentals of statistical testing 5237.1.1 The p-value and its interpretation 5257.1.2 Ad hoc statistical testing 5287.2 Simple hypothesis 5317.3 The power function of the Z-test 5367.3.1 Type II error and the power function 5367.3.2 Optimal significance level and the ROC curve 5427.3.3 One-sided hypothesis 5457.4 The t-test for the means 5497.4.1 One-sample t-test 5497.4.2 Two-sample t-test 5527.4.3 One-sided t-test 5577.4.4 Paired versus unpaired t-test 5587.4.5 Parametric versus nonparametric tests 5607.5 Variance test 5627.5.1 Two-sided variance test 5627.5.2 One-sided variance test 5657.6 Inverse-cdf test 5667.6.1 General formulation 5677.6.2 The F-test for variances 5697.6.3 Binomial proportion 5737.6.4 Poisson rate 5777.7 Testing for correlation coefficient 5807.8 Confidence interval 5837.8.1 Unbiased CI and its connection to hypothesis testing 5887.8.2 Inverse cdf CI 5897.8.3 CI for the normal variance and SD 5917.8.4 CI for other major statistical parameters 5927.8.5 Confidence region 5947.9 Three asymptotic tests and confidence intervals 5977.9.1 Pearson chi-square test 6057.9.2 Handwritten digit recognition 6087.10 Limitations of classical hypothesis testing and the d-value 6127.10.1 What the p-value means? 6137.10.2 Why   = 0.05? 6147.10.3 The null hypothesis is always rejected with a large enough sample size 6167.10.4 Parameter-based inference 6187.10.5 The d-value for individual inference 6198 Linear model and its extensions 6278.1 Basic definitions and linear least squares 6278.1.1 Linear model with the intercept term 6328.1.2 The vector-space geometry of least squares 6338.1.3 Coefficient of determination 6368.2 The Gauss-Markov theorem 6398.2.1 Estimation of regression variance 6418.3 Properties of OLS estimators under the normal assumption 6438.3.1 The sensitivity of statistical inference to violation of the normal assumption 6468.4 Statistical inference with linear models 6508.4.1 Confidence interval and region 6508.4.2 Linear hypothesis testing and the F-test 6538.4.3 Prediction by linear regression and simultaneous confidence band 6618.4.4 Testing the null hypothesis and the coefficient of determination 6648.4.5 Is X fixed or random? 6658.5 The one-sided p- and d-value for regression coefficients 6718.5.1 The one-sided p-value for interpretation on the population level 6728.5.2 The d-value for interpretation on the individual level 6738.6 Examples and pitfalls 6768.6.1 Kids drinking and alcohol movie watching 6768.6.2 My first false discovery 6808.6.3 Height, foot, and nose regression 6818.6.4 A geometric interpretation of adding a new predictor 6848.6.5 Contrast coefficient of determination against spurious regression 6878.7 Dummy variable approach and ANOVA 6968.7.1 Dummy variables for categories 6968.7.2 Unpaired and paired t-test 7058.7.3 Modeling longitudinal data 7088.7.4 One-way ANOVA model 7128.7.5 Two-way ANOVA 7208.8 Generalized linear model 7238.8.1 MLE estimation of GLM 7278.8.2 Logistic and probit regressions for binary outcome 7288.8.3 Poisson regression 7369 Nonlinear regression 7419.1 Definition and motivating examples 7419.2 Nonlinear least squares 7509.3 Gauss-Newton algorithm 7539.4 Statistical properties of the NLS estimator 7579.4.1 Large sample properties 7579.4.2 Small sample properties 7629.4.3 Asymptotic confidence intervals and hypothesis testing 7639.4.4 Three methods of statistical inference in large sample 7689.5 The nls function and examples 7709.5.1 NLS-cdf estimator 7829.6 Studying small sample properties through simulations 7869.6.1 Normal distribution approximation 7879.6.2 Statistical tests 7899.6.3 Confidence region 7919.6.4 Confidence intervals 7929.7 Numerical complications of the nonlinear least squares 7949.7.1 Criteria for existence 7959.7.2 Criteria for uniqueness 7969.8 Optimal design of experiments with nonlinear regression 7999.8.1 Motivating examples 7999.8.2 Optimal designs with nonlinear regression 8029.9 The Michaelis-Menten model 8059.9.1 The NLS solution 8069.9.2 The exact solution 80710 Appendix 81110.1 Notation 81110.2 Basics of matrix algebra 81110.2.1 Preliminaries and matrix inverse 81210.2.2 Determinant 81510.2.3 Partition matrices 81610.3 Eigenvalues and eigenvectors 81810.3.1 Jordan spectral matrix decomposition 81910.3.2 SVD: Singular value decomposition of a rectangular matrix 82010.4 Quadratic forms and positive definite matrices 82210.4.1 Quadratic forms 82210.4.2 Positive and nonnegative definite matrices 82310.5 Vector and matrix calculus 82610.5.1 Differentiation of a scalar-valued function with respect to a vector 82610.5.2 Differentiation of a vector-valued function with respect to a vector 82710.5.3 Kronecker product 82810.5.4 vec operator 82810.6 Optimization 82910.6.1 Convex and concave functions 83010.6.2 Criteria for unconstrained minimization 83110.6.3 Gradient algorithms 83510.6.4 Constrained optimization: Lagrange multiplier technique 838Bibliography 843Index 851