Head First Data Analysis

Author of Head First Data Analysis......Page 10
Table of Contents (the real thing)......Page 11
how to use this book: Intro......Page 29
Who should probably back away from this book?......Page 30
We know what your brain is thinking......Page 31
Here’s what WE did:......Page 34
Here’s what YOU can do to bend your brain into submission......Page 35
Read Me......Page 36
The technical review team......Page 38
Acknowledgments......Page 39
Safari® Books Online......Page 40
1 introduction to data analysis: Break it down......Page 41
Acme Cosmetics needs your help......Page 42
The CEO wants data analysis to help increase sales......Page 43
Data analysis is careful thinking about evidence......Page 44
Define the problem......Page 45
Your client will help you define your problem......Page 46
Acme’s CEO has some feedback for you......Page 48
Break the problem and data into smaller pieces......Page 49
Now take another look at what you know......Page 50
Evaluate the pieces......Page 53
Analysis begins when you insert yourself......Page 54
Make a recommendation......Page 55
Your report is ready......Page 56
The CEO likes your work......Page 57
An article just came across the wire......Page 58
You let the CEO’s beliefs take you down the wrong path......Page 60
Your assumptions and beliefs about the world are your mental model......Page 61
Your statistical model depends on your mental model......Page 62
Mental models should always include what you don’t know......Page 65
The CEO tells you what he doesn’t know......Page 66
Acme just sent you a huge list of raw data......Page 68
Time to drill further into the data......Page 71
General American Wholesalers confirms your impression......Page 72
Here’s what you did......Page 75
Your analysis led your client to a brilliant decision......Page 76
2 experiments: Test your theories......Page 77
It’s a coffee recession!......Page 78
The Starbuzz boardmeeting is in three months......Page 79
The Starbuzz Survey......Page 81
Always use the method of comparison......Page 82
Comparisons are key for observational data......Page 83
Could value perception be causing the revenue decline?......Page 84
A typical customer’s thinking......Page 86
Observational studies are full of confounders......Page 87
How location might be confounding your results......Page 88
Manage confounders by breaking the data into chunks......Page 90
It’s worse than we thought!......Page 93
You need an experiment to say which strategy will work best......Page 94
The Starbuzz CEO is in a big hurry......Page 95
Starbuzz drops its prices......Page 96
One month later…......Page 97
Control groups give you a baseline......Page 98
Not getting fired 101......Page 101
Let’s experiment again for real!......Page 102
One month later…......Page 103
Confounders also plague experiments......Page 104
Avoid confounders by selecting groups carefully......Page 105
Randomization selects similar groups......Page 107
Your experiment is ready to go......Page 111
The results are in......Page 112
Starbuzz has an empirically tested sales strategy......Page 113
3 optimization: Take it to the max......Page 115
You’re now in the bath toy game......Page 116
Decision variables are things you can control......Page 119
You have an optimization problem......Page 120
Find your objective with the objective function......Page 121
Your objective function......Page 122
Show product mixes with your other constraints......Page 123
Plot multiple constraints on the same chart......Page 124
Your good options are all in the feasible region......Page 125
Your new constraint changed the feasible region......Page 127
Your spreadsheet does optimization......Page 130
Solver crunched your optimization problem in a snap......Page 134
Profits fell through the floor......Page 137
Your model only describes what you put into it......Page 138
Calibrate your assumptions to your analytical objectives......Page 139
Watch out for negatively linked variables......Page 143
Your new plan is working like a charm......Page 148
Your assumptions are based on an ever-changing reality......Page 149
4 data visualization: Pictures make you smarter......Page 151
New Army needs to optimize their website......Page 152
The results are in, but the information designer is out......Page 153
The last information designer submitted these three infographics......Page 154
What data is behind the visualizations?......Page 155
Show the data!......Page 156
Here’s some unsolicited advice from the last designer......Page 157
Too much data is never your problem......Page 158
Making the data pretty isn’t your problem either......Page 159
Data visualization is all about making the right comparisons......Page 160
Your visualization is already more useful than the rejected ones......Page 163
Use scatterplots to explore causes......Page 164
The best visualizations are highly multivariate......Page 165
Show more variables by looking at charts together......Page 166
The visualization is great, but the web guru’s not satisfied yet......Page 170
Good visual designs help you think about causes......Page 171
The experiment designers weigh in......Page 172
The experiment designers have some hypotheses of their own......Page 175
The client is pleased with your work......Page 176
Orders are coming in from everywhere!......Page 177
5 hypothesis testing: Say it ain’t so......Page 179
Gimme some skin…......Page 180
When do we start making new phone skins?......Page 181
PodPhone doesn’t want you to predict their next move......Page 182
Here’s everything we know......Page 183
ElectroSkinny’s analysis does fit the data......Page 184
ElectroSkinny obtained this confidential strategy memo......Page 185
Variables can be negatively or positively linked......Page 186
Causes in the real world are networked, not linear......Page 189
Hypothesize PodPhone’s options......Page 190
You have what you need to run a hypothesis test......Page 191
Falsification is the heart of hypothesis testing......Page 192
Diagnosticity helps you find the hypothesis with the least disconfirmation......Page 200
You can’t rule out all the hypotheses,but you can say which is strongest......Page 203
You just got a picture message…......Page 204
It’s a launch!......Page 207
6 bayesian statistics: Get past first base......Page 209
The doctor has disturbing news......Page 210
Let’s take the accuracy analysis one claim at a time......Page 213
How common is lizard flu really?......Page 214
You’ve been counting false positives......Page 215
All these terms describe conditional probabilities......Page 216
You need to count false positives, true positives, false negatives, and true negatives......Page 217
1 percent of people have lizard flu......Page 218
Your chances of having lizard flu are still pretty low......Page 221
Bayes’ rule manages your base rates when you get new data......Page 222
You can use Bayes’ rule over and over......Page 223
Your second test result is negative......Page 224
The new test has different accuracy statistics......Page 225
New information can change your base rate......Page 226
What a relief!......Page 229
7 subjective probabilities: Numerical belief......Page 231
Backwater Investments needs your help......Page 232
Their analysts are at each other’s throats......Page 233
Subjective probabilities describe expert beliefs......Page 238
Subjective probabilities might show no real disagreement after all......Page 239
The analysts responded with their subjective probabilities......Page 241
The CEO doesn’t see what you’re up to......Page 242
The CEO loves your work......Page 247
The standard deviation measures how far points are from the average......Page 248
You were totally blindsided by this news......Page 253
Bayes’ rule is great for revising subjective probabilities......Page 257
The CEO knows exactly what to do with this new information......Page 263
Russian stock owners rejoice!......Page 264
8 heuristics: Analyze like a human......Page 265
LitterGitters submitted their report to the city council......Page 266
The LitterGitters have really cleaned up this town......Page 267
The LitterGitters have been measuring their campaign’s effectiveness......Page 268
The mandate is to reduce the tonnage of litter......Page 269
Tonnage is unfeasible to measure......Page 270
Give people a hard question, and they’ll answer an easier one instead......Page 271
Littering in Dataville is a complex system......Page 272
You can’t build and implement a unified litter-measuring model......Page 273
Heuristics are a middle ground between going with your gut and optimization......Page 276
Use a fast and frugal tree......Page 279
Is there a simpler way to assess LitterGitters’ success?......Page 280
Stereotypes are heuristics......Page 284
Your analysis is ready to present......Page 286
Looks like your analysis impressed the city council members......Page 289
9 histograms: The shape of numbers......Page 291
Your annual review is coming up......Page 292
Going for more cash could play out in a bunch of different ways......Page 294
Here’s some data on raises......Page 295
Histograms show frequencies of groups of numbers......Page 302
Gaps between bars in a histogram mean gaps among the data points......Page 303
Install and run R......Page 304
Load data into R......Page 305
R creates beautiful histograms......Page 306
Make histograms from subsets of your data......Page 311
Negotiation pays......Page 316
What will negotiation mean for you?......Page 317
10 regression: Prediction......Page 319
What are you going to do with all this money?......Page 320
An analysis that tells people what to ask for could be huge......Page 323
Behold… the Raise Reckoner!......Page 324
Inside the algorithm will be a method to predict raises......Page 326
Scatterplots compare two variables......Page 332
A line could tell your clients where to aim......Page 334
Predict values in each strip with the graph of averages......Page 337
The regression line predicts what raises people will receive......Page 338
The line is useful if your data shows a linear correlation......Page 340
You need an equation to make your predictions precise......Page 344
Tell R to create a regression object......Page 346
The regression equation goes hand in hand with your scatterplot......Page 349
The regression equation is the Raise Reckoner algorithm......Page 350
Your raise predictor didn’t work out as planned…......Page 353
11 error: Err Well......Page 355
Your clients are pretty ticked off......Page 356
What did your raise prediction algorithm do?......Page 357
The segments of customers......Page 358
The guy who asked for 25%went outside the model......Page 361
How to handle the client who wants a prediction outside the data range......Page 362
The guy who got fired because of extrapolation has cooled off......Page 367
You’ve only solved part of the problem......Page 368
What does the data for the screwy outcomes look like?......Page 369
Chance errors are deviations from what your model predicts......Page 370
Error is good for you and your client......Page 374
Specify error quantitatively......Page 376
Quantify your residual distribution with Root Mean Squared error......Page 377
Your model in R already knows the R.M.S. error......Page 378
R’s summary of your linear model shows your R.M.S. error......Page 380
Segmentation is all about managing error......Page 386
Good regressions balance explanation and prediction......Page 390
Your segmented models manage error better than the original model......Page 392
Your clients are returning in droves......Page 397
12 relational databases: Can you relate?......Page 399
The Dataville Dispatch wants to analyze sales......Page 400
Here’s the data they keep to track their operations......Page 401
You need to know how the data tables relate to each other......Page 402
A database is a collection of data with well‑specified relations to each other......Page 405
Create a spreadsheet that goes across that path......Page 406
Your summary ties article count and sales together......Page 411
Looks like your scatterplot is going over really well......Page 414
Copying and pasting all that data was a pain......Page 415
Relational databases manage relations for you......Page 416
Dataville Dispatch built an RDBMS with your relationship diagram......Page 417
Dataville Dispatch extracted your data using the SQL language......Page 419
Comparison possibilities are endless if your data is in a RDBMS......Page 422
You’re on the cover......Page 423
13 cleaning data: Impose order......Page 425
Just got a client list from a defunct competitor......Page 426
The dirty secret of data analysis......Page 427
Head First Head Hunters wants the list for their sales team......Page 428
Cleaning messy data is all about preparation......Page 432
Once you’re organized, you can fix the data itself......Page 433
Use the # sign as a delimiter......Page 434
Excel split your data into columns using the delimiter......Page 435
Use SUBSTITUTE to replace the carat character......Page 439
You cleaned up all the first names......Page 440
The last name pattern is too complex for SUBSTITUTE......Page 442
Handle complex patterns with nested text formulas......Page 443
R can use regular expressions to crunch complex data patterns......Page 444
The sub command fixed your last names......Page 446
Now you can ship the data to your client......Page 447
Maybe you’re not quite done yet…......Page 448
Sort your data to show duplicate values together......Page 449
The data is probably from a relational database......Page 452
Remove duplicate names......Page 453
You created nice, clean, unique records......Page 454
Head First Head Hunters is recruiting like gangbusters!......Page 455
Leaving town.........Page 456
appendix i: leftovers: The Top Ten Things (we didn\'t cover)......Page 457
#1: Everything else in statistics......Page 458
#2: Excel skills......Page 459
#3: Edward Tufte and his principles of visualization......Page 460
#4: PivotTables......Page 461
#5: The R community......Page 462
#6: Nonlinear and multiple regression......Page 463
#8: Randomness......Page 464
#9: Google Docs......Page 465
#10: Your expertise......Page 466
appendix ii: install r: Start R up!......Page 467
Get started with R......Page 468
appendix iii: install excel analysis tools:The ToolPak......Page 471
Install the data analysis tools in Excel......Page 472
B......Page 475
C......Page 476
D......Page 477
E......Page 478
I......Page 479
O......Page 480
Q......Page 481
R......Page 482
S......Page 483
V......Page 484
Y......Page 485