Probability and statistics for data science : math + R + data

Cover......Page 1
Half Title......Page 2
Title Page......Page 4
Copyright Page......Page 5
Table of Contents......Page 6
About the Author......Page 24
To the Instructor......Page 26
To the Reader......Page 32
I: Fundamentals of Probability......Page 34
  1.1 Example: Bus Ridership......Page 36
  1.2 A \" Notebook\" View: the Notion of a Repeatable Experiment......Page 37
   1.2.2 A More Intuitive Approach......Page 38
  1.3 Our Definitions......Page 40
  1.5 Example: Bus Ridership Model (cont\'d.)......Page 44
  1.6 Example: ALOHA Network......Page 47
   1.6.2 ALOHA Network Computations......Page 49
  1.7 ALOHA in the Notebook Context......Page 52
  1.8 Example: A Simple Board Game......Page 53
   1.9.2 Example: Document Classification......Page 56
  1.10 Random Graph Models......Page 57
   1.10.1 Example: Preferential Attachment Model......Page 58
   1.11.1 Which Is More Likely in Five Cards, One King or Two Hearts?......Page 59
   1.11.3 Example: Lottery Tickets......Page 60
   1.11.4 Example: Gaps between Numbers......Page 61
   1.11.5 Multinomial Coeffcients......Page 62
   1.11.6 Example: Probability of Getting Four Aces in a Bridge Hand......Page 63
  1.12 Exercises......Page 64
  2.1 Example: Rolling Dice......Page 68
   2.1.1 First Improvement......Page 69
   2.1.2 Second Improvement......Page 70
   2.1.3 Third Improvement......Page 71
  2.3 Use of runif() for Simulating Events......Page 72
  2.5 Example: Board Game (cont\'d.)......Page 73
  2.6 Example: Broken Rod......Page 74
   2.8.1 More on the replicate() Function......Page 75
  2.9 Exercises......Page 76
  3.1 Random Variables......Page 78
  3.3 Independent Random Variables......Page 79
  3.4 Example: The Monty Hall Problem......Page 80
   3.5.3 Definition and Notebook View......Page 83
   3.6.1 Computational Formula......Page 84
   3.6.2 Further Properties of Expected Value......Page 87
  3.8 Example: Predicting Product Demand......Page 91
  3.9 Expected Values via Simulation......Page 92
  3.10 Casinos, Insurance Companies and \"Sum Users,\" Compared to Others......Page 93
   3.11.1 Proof of Property E......Page 94
  3.12 Exercises......Page 95
   4.1.1 Definition......Page 98
    4.1.3.1 Chebychev\'s Inequality......Page 102
    4.1.3.2 The Coecient of Variation......Page 103
  4.2 A Useful Fact......Page 104
  4.3 Covariance......Page 105
  4.4 Indicator Random Variables, and Their Means and Variances......Page 107
   4.4.1 Example: Return Time for Library Books, Version I......Page 108
   4.4.2 Example: Return Time for Library Books, Version II......Page 109
   4.4.3 Example: Indicator Variables in a Committee Problem......Page 110
   4.6.1 Proof of Chebychev\'s Inequality......Page 112
  4.7 Exercises......Page 114
  5.1 Distributions......Page 116
   5.1.1 Example: Toss Coin Until First Head......Page 117
    5.1.3.1 The Model......Page 118
  5.3 The Case of Importance to Us: Parameteric Families of pmfs......Page 119
   5.4.1 The Geometric Family of Distributions......Page 121
    5.4.1.1 R Functions......Page 124
    5.4.1.2 Example: A Parking Space Problem......Page 125
   5.4.2 The Binomial Family of Distributions......Page 127
    5.4.2.1 R Functions......Page 128
   5.4.3 The Negative Binomial Family of Distributions......Page 129
    5.4.3.1 R Functions......Page 130
  5.5 Two Major Non-Bernoulli Models......Page 131
    5.5.1.1 R Functions......Page 132
    5.5.2.1 The Model......Page 133
    5.5.3.1 Poisson Model......Page 135
    5.5.3.3 Example: DNC E-mail Data......Page 136
   5.6.1 Example: The Bus Ridership Problem......Page 139
   5.6.2 Example: Analysis of Social Networks......Page 140
   5.7.1 Graphics and Visualization in R......Page 141
  5.8 Exercises......Page 142
  6.1 A Random Dart......Page 146
  6.2 Individual Values Now Have Probability Zero......Page 147
   6.4.1 CDFs......Page 148
  6.5 Density Functions......Page 152
   6.5.1 Properties of Densities......Page 153
   6.5.3 Expected Values......Page 155
  6.6 A First Example......Page 156
  6.7 Famous Parametric Families of Continuous Distributions......Page 157
    6.7.1.2 R Functions......Page 158
    6.7.1.4 Example: Modeling of Denial-of-Service Attack......Page 159
    6.7.2.2 R Functions......Page 160
    6.7.3.2 R Functions......Page 161
    6.7.3.3 Example: Garage Parking Fees......Page 162
    6.7.3.4 Memoryless Property of Exponential Distributions......Page 163
   6.7.4 The Gamma Family of Distributions......Page 164
    6.7.4.1 Density and Properties......Page 165
    6.7.4.3 Importance in Modeling......Page 166
    6.7.5.1 Density Etc.......Page 167
   6.8.1 Hazard Functions......Page 171
   6.8.2 Duality of the Exponential Family with the Poisson Family......Page 172
   6.9.2 Inverse Method for Sampling from a Density......Page 174
   6.9.3 Sampling from a Poisson Distribution......Page 175
  6.10 Exercises......Page 176
II: Fundamentals of Statistics......Page 180
 7: Statistics: Prologue......Page 182
   7.2.1 Random Samples......Page 183
   7.3.1 Toy Population Example......Page 185
   7.3.2 Expected Value and Variance of x̄......Page 186
   7.3.3 Toy Population Example Again......Page 187
   7.3.5 Notebook View......Page 188
  7.4 Simple Random Sample Case......Page 189
   7.5.1 Intuitive Estimation of σ2......Page 190
   7.5.3 Special Case: X Is an Indicator Variable......Page 191
   7.6.1 Statistical Bias......Page 192
  7.7 The Concept of a \"Standard Error\"......Page 194
  7.8 Example: Pima Diabetes Study......Page 195
   7.10.2 Infinite Populations?......Page 197
   7.12.1 The *apply() Functions......Page 198
    7.12.1.2 The lapply() and sapply() Function......Page 199
    7.12.1.3 The split() and tapply() Functions......Page 200
   7.12.2 Outliers/Errors in the Data......Page 201
  7.13 Exercises......Page 203
  8.1 Why Fit a Parametric Model?......Page 204
   8.2.1 A Closer Look......Page 205
   8.2.2 Example: BMI Data......Page 206
   8.2.3 The Number of Bins......Page 207
    8.2.3.1 The Bias-Variance Tradeoff......Page 208
    8.2.3.2 The Bias-Variance Tradeoff in the Histogram Case......Page 209
    8.2.3.3 A General Issue: Choosing the Degree of Smoothing......Page 211
  8.3 Advanced Methods for Model-Free Density Estimation......Page 213
   8.4.1 Method of Moments......Page 214
   8.4.2 Example: BMI Data......Page 215
   8.4.3 The Method of Maximum Likelihood......Page 216
   8.4.4 Example: Humidity Data......Page 218
  8.6 Assessment of Goodness of Fit......Page 220
  8.7 The Bayesian Philosophy......Page 222
   8.7.2 Arguments For and Against......Page 223
   8.8.1 Details of Kernel Density Estimators......Page 224
   8.9.1 Generic Functions......Page 225
    8.9.2.2 Example: Bodyfat Data......Page 226
  8.10 Exercises......Page 227
  9.1 Density and Properties......Page 230
   9.1.1 Closure under Affine Transformation......Page 231
   9.1.2 Closure under Independent Summation......Page 232
  9.3 The Standard Normal Distribution......Page 233
  9.4 Evaluating Normal cdfs......Page 234
  9.5 Example: Network Intrusion......Page 235
  9.6 Example: Class Enrollment Size......Page 236
  9.7 The Central Limit Theorem......Page 237
   9.7.2 Example: Coin Tosses......Page 238
   9.7.3 Example: Museum Demonstration......Page 239
   9.8.1 Approximate Distribution of X......Page 240
   9.8.2 Improved Assessment of Accuracy of X......Page 241
  9.9 Importance in Modeling......Page 242
   9.10.1 Density and Properties......Page 243
   9.10.3 Importance in Modeling......Page 244
   9.11.1 Convergence in Distribution, and the Precisely-Stated CLT......Page 245
   9.12.1 Example: Generating Normal Random Numbers......Page 246
  9.13 Exercises......Page 247
  10.1 The Role of Normal Distributions......Page 250
   10.2.1 Basic Formulation......Page 251
  10.3 Example: Pima Diabetes Study......Page 253
   10.5.1 A Weight Survey in Davis......Page 254
  10.6 Confidence Intervals for Proportions......Page 256
   10.6.1 Example: Machine Classification of Forest Covers......Page 257
  10.7 The Student-t Distribution......Page 259
  10.8 Introduction to Significance Tests......Page 260
  10.9 The Proverbial Fair Coin......Page 261
  10.10 The Basics......Page 262
  10.12 The Notion of \"p-Values\"......Page 264
  10.14 Example: The Forest Cover Data......Page 265
   10.15.1 History of Significance Testing......Page 267
   10.15.2 The Basic Issues......Page 268
   10.15.3 Alternative Approach......Page 269
  10.16 The Problem of \"P-hacking\"......Page 270
   10.16.2 Multiple Inference Methods......Page 271
   10.17.1 More about Interpretation of CIs......Page 272
  10.18 Exercises......Page 274
III: Multivariate Analysis......Page 276
   11.1.1 Example: Marbles in a Bag......Page 278
   11.2.1 Motivation and Definition......Page 279
   11.2.3 Example: Train Rendezvous......Page 280
   11.3.1 Covariance......Page 281
   11.3.2 Example: The Committee Example Again......Page 283
  11.4 Correlation......Page 284
   11.5.1 Mailing Tubes......Page 285
    11.5.1.3 Variances Add......Page 286
   11.6.2 Covariance Matrices......Page 287
   11.6.3 Mailing Tubes: Covariance Matrices......Page 288
  11.7 Sample Estimate of Covariance Matrix......Page 289
   11.8.1 Convolution......Page 290
    11.8.1.1 Example: Backup Battery......Page 291
    11.8.2.1 Generating Functions......Page 292
    11.8.2.2 Sums of Independent Poisson Random Variables Are Poisson Distributed......Page 294
  11.9 Exercises......Page 295
  12.1 Densities......Page 298
  12.2 Geometric Interpretation......Page 299
  12.3 R Functions......Page 302
  12.5 Properties of Multivariate Normal Distributions......Page 303
  12.6 The Multivariate Central Limit Theorem......Page 305
  12.7 Exercises......Page 306
 13: Mixture Distributions......Page 308
  13.1 Iterated Expectations......Page 309
   13.1.2 The Theorem......Page 310
   13.1.3 Example: Flipping Coins with Bonuses......Page 312
   13.1.5 What about Variance?......Page 313
   13.2.1 Derivation of Mean and Variance......Page 314
    13.2.2.1 Example: Old Faithful Estimation......Page 316
  13.3 Clustering......Page 317
  13.4 Exercises......Page 318
 14: Multivariate Description and Dimension Reduction......Page 320
   14.1.1 \"Desperate for Data\"......Page 321
   14.1.3 Estimated Mean......Page 322
   14.1.4 The Bias/Variance Tradeoff: Concrete Illustration......Page 323
   14.1.5 Implications......Page 325
   14.2.1 Intuition......Page 326
   14.2.2 Properties of PCA......Page 328
   14.2.3 Example: Turkish Teaching Evaluations......Page 329
   14.3.1 Example: Hair Color, Eye Color and Gender......Page 330
   14.3.3 Estimating the Parameters......Page 332
   14.4.1 Statistical Derivation of PCA......Page 333
   14.5.2 Some Details on Log-Linear Models......Page 335
    14.5.2.1 Parameter Estimation......Page 336
    14.5.2.2 The loglin() Function......Page 337
    14.5.2.3 Informal Assessment of Fit......Page 338
  14.6 Exercises......Page 339
  15.1 Example: Heritage Health Prize......Page 342
   15.2.1 Terminology......Page 343
   15.3.1 Precise Definition......Page 344
   15.3.2 Parametric Models for the Regression Function m()......Page 346
  15.4 Estimation in Linear Parametric Regression Models......Page 347
  15.5 Example: Baseball Data......Page 348
   15.5.1 R Code......Page 349
  15.6 Multiple Regression......Page 352
  15.7 Example: Baseball Data (cont\'d.)......Page 353
  15.8 Interaction Terms......Page 354
   15.9.2 Random-X and Fixed-X Regression......Page 355
   15.9.3 Point Estimates and Matrix Formulation......Page 356
   15.9.4 Approximate Con dence Intervals......Page 359
  15.10 Example: Baseball Data (cont\'d.)......Page 361
  15.11 Dummy Variables......Page 362
  15.12 Classification......Page 363
   15.12.1 Classification = Regression......Page 364
    15.12.2.1 The Logistic Model: Motivations......Page 365
   15.12.4 R Code......Page 367
   15.12.5 Analysis of the Results......Page 368
   15.13.1 Example: Predicting Vertebral Abnormalities......Page 369
   15.13.3 R Packages......Page 372
   15.14.1 Computational Details in Section 15.5.1......Page 373
   15.14.2 More Regarding glm()......Page 374
  15.15 Exercises......Page 375
   16.1.1 Example: Histograms......Page 376
   16.1.2 Example: Polynomial Regression......Page 377
   16.2.1 Cross-Validation......Page 378
  16.3 Predictor Subset Selection......Page 379
  16.4 Exercises......Page 380
 17: Introduction to Discrete Time Markov Chains......Page 382
  17.1 Matrix Formulation......Page 383
  17.2 Example: Die Game......Page 384
  17.3 Long-Run State Probabilities......Page 385
   17.3.1 Stationary Distribution......Page 386
   17.3.2 Calculation of π......Page 387
   17.3.3 Simulation Calculation of π......Page 388
  17.4 Example: 3-Heads-in-a-Row Game......Page 389
  17.5 Example: Bus Ridership Problem......Page 391
  17.6 Hidden Markov Models......Page 392
   17.6.1 Example: Bus Ridership......Page 393
   17.8.1 Initializing a Matrix to All 0s......Page 394
  17.9 Exercises......Page 395
IV: Appendices......Page 398
  A.1 Starting R......Page 400
  A.2 Correspondences......Page 401
  A.3 First Sample Programming Session......Page 402
  A.5 Second Sample Programming Session......Page 405
  A.7 More on Vectorization......Page 407
  A.8 Default Argument Values......Page 408
   A.9.1 The Basics......Page 409
   A.9.2 S3 Classes......Page 410
  A.10 Data Frames......Page 411
  A.12 Debugging in R......Page 413
   B.1.1 Matrix Addition and Multiplication......Page 416
  B.4 Eigenvalues and Eigenvectors......Page 418
   B.5.1 Matrix Derivatives......Page 419
Bibliography......Page 424
Index......Page 428