Principles and Practice of Big Data: Preparing, Sharing, and Analyzing Complex Information

Preface......Page 3
Section 1.1. Definition of Big Data......Page 11
Section 1.2. Big Data Versus Small Data......Page 13
Section 1.3. Whence Comest Big Data?......Page 15
Section 1.4. The Most Common Purpose of Big Data Is to Produce Small Data......Page 17
Section 1.5. Big Data Sits at the Center of the Research Universe......Page 18
Glossary......Page 19
References......Page 23
Section 2.1. Nearly All Data Is Unstructured and Unusable in Its Raw Form......Page 24
Section 2.2. Concordances......Page 25
Section 2.3. Term Extraction......Page 28
Section 2.4. Indexing......Page 31
Section 2.5. Autocoding......Page 33
Section 2.6. Case Study: Instantly Finding the Precise Location of Any Atom in the Universe (Some Assembly Required)......Page 38
Section 2.7. Case Study (Advanced): A Complete Autocoder (in 12 Lines of Python Code)......Page 40
Section 2.8. Case Study: Concordances as Transformations of Text......Page 43
Section 2.9. Case Study (Advanced): Burrows Wheeler Transform (BWT)......Page 45
Glossary......Page 48
References......Page 59
Section 3.1. What Are Identifiers?......Page 61
Section 3.2. Difference Between an Identifier and an Identifier System......Page 63
Section 3.3. Generating Unique Identifiers......Page 66
Section 3.4. Really Bad Identifier Methods......Page 68
Section 3.5. Registering Unique Object Identifiers......Page 71
Section 3.6. Deidentification and Reidentification......Page 74
Section 3.7. Case Study: Data Scrubbing......Page 77
Section 3.8. Case Study (Advanced): Identifiers in Image Headers......Page 79
Section 3.9. Case Study: One-Way Hashes......Page 82
Glossary......Page 84
References......Page 90
Section 4.2. eXtensible Markup Language......Page 93
Section 4.3. Semantics and Triples......Page 95
Section 4.4. Namespaces......Page 96
Section 4.5. Case Study: A Syntax for Triples......Page 98
Section 4.6. Case Study: Dublin Core......Page 101
Glossary......Page 102
References......Page 103
Section 5.1. It’s All About Object Relationships......Page 104
Section 5.2. Classifications, the Simplest of Ontologies......Page 108
Section 5.3. Ontologies, Classes With Multiple Parents......Page 111
Section 5.4. Choosing a Class Model......Page 113
Section 5.5. Class Blending......Page 117
Section 5.6. Common Pitfalls in Ontology Development......Page 118
Section 5.7. Case Study: An Upper Level Ontology......Page 121
Section 5.8. Case Study (Advanced): Paradoxes......Page 122
Section 5.9. Case Study (Advanced): RDF Schemas and Class Properties......Page 124
Section 5.10. Case Study (Advanced): Visualizing Class Relationships......Page 127
Glossary......Page 132
References......Page 141
Section 6.1. Knowledge of Self......Page 143
Section 6.2. Data Objects: The Essential Ingredient of Every Big Data Collection......Page 146
Section 6.3. How Big Data Uses Introspection......Page 148
Section 6.4. Case Study: Time Stamping Data......Page 151
Section 6.5. Case Study: A Visit to the TripleStore......Page 153
Section 6.6. Case Study (Advanced): Proof That Big Data Must Be Object-Oriented......Page 158
Glossary......Page 159
References......Page 160
Section 7.1. Standards......Page 161
Section 7.2. Specifications Versus Standards......Page 166
Section 7.3. Versioning......Page 168
Section 7.4. Compliance Issues......Page 170
Section 7.5. Case Study: Standardizing the Chocolate Teapot......Page 171
Glossary......Page 172
References......Page 173
Section 8.1. The Importance of Data That Cannot Change......Page 174
Section 8.2. Immutability and Identifiers......Page 175
Section 8.3. Coping With the Data That Data Creates......Page 178
Section 8.4. Reconciling Identifiers Across Institutions......Page 179
Section 8.6. Case Study: Blockchains and Distributed Ledgers......Page 181
Section 8.7. Case Study (Advanced): Zero-Knowledge Reconciliation......Page 184
Glossary......Page 186
References......Page 188
Section 9.1. Looking at the Data......Page 189
Section 9.2. The Minimal Necessary Properties of Big Data......Page 196
Section 9.3. Data That Comes With Conditions......Page 201
Section 9.4. Case Study: Utilities for Viewing and Searching Large Files......Page 202
Section 9.5. Case Study: Flattened Data......Page 204
Glossary......Page 205
References......Page 209
Section 10.1. Accuracy and Precision......Page 211
Section 10.2. Data Range......Page 213
Section 10.3. Counting......Page 215
Section 10.4. Normalizing and Transforming Your Data......Page 219
Section 10.5. Reducing Your Data......Page 223
Section 10.6. Understanding Your Control......Page 226
Section 10.7. Statistical Significance Without Practical Significance......Page 227
Section 10.8. Case Study: Gene Counting......Page 228
Section 10.9. Case Study: Early Biometrics, and the Significance of Narrow Data Ranges......Page 229
Glossary......Page 230
References......Page 232
Section 11.1. Speed and Scalability......Page 234
Section 11.2. Fast Operations, Suitable for Big Data, That Every Computer Supports......Page 240
Section 11.3. The Dot Product, a Simple and Fast Correlation Method......Page 246
Section 11.4. Clustering......Page 248
Section 11.5. Methods for Data Persistence (Without Using a Database)......Page 250
Section 11.6. Case Study: Climbing a Classification......Page 252
Section 11.7. Case Study (Advanced): A Database Example......Page 254
Section 11.8. Case Study (Advanced): NoSQL......Page 255
Glossary......Page 256
References......Page 259
Section 12.1. Denominators......Page 261
Section 12.2. Word Frequency Distributions......Page 262
Section 12.3. Outliers and Anomalies......Page 266
Section 12.4. Back-of-Envelope Analyses......Page 268
Section 12.5. Case Study: Predicting User Preferences......Page 270
Section 12.6. Case Study: Multimodality in Population Data......Page 272
Glossary......Page 273
References......Page 277
Section 13.1. The Remarkable Utility of (Pseudo)Random Numbers......Page 278
Section 13.2. Repeated Sampling......Page 284
Section 13.3. Monte Carlo Simulations......Page 289
Section 13.4. Case Study: Proving the Central Limit Theorem......Page 292
Section 13.5. Case Study: Frequency of Unlikely String of Occurrences......Page 294
Section 13.6. Case Study: The Infamous Birthday Problem......Page 295
Section 13.7. Case Study (Advanced): The Monty Hall Problem......Page 296
Section 13.8. Case Study (Advanced): A Bayesian Analysis......Page 298
Glossary......Page 300
References......Page 302
Section 14.1. Theory in Search of Data......Page 304
Section 14.2. Data in Search of Theory......Page 305
Section 14.3. Bigness Biases......Page 306
Section 14.4. Data Subsets in Big Data: Neither Additive Nor Transitive......Page 311
Section 14.5. Additional Big Data Pitfalls......Page 312
Section 14.6. Case Study (Advanced): Curse of Dimensionality......Page 315
Glossary......Page 317
References......Page 319
Section 15.1. Failure Is Common......Page 321
Section 15.2. Failed Standards......Page 323
Section 15.3. Blaming Complexity......Page 326
Section 15.4. An Approach to Big Data That May Work for You......Page 328
Section 15.5. After Failure......Page 337
Section 15.6. Case Study: Cancer Biomedical Informatics Grid, a Bridge Too Far......Page 339
Section 15.7. Case Study: The Gaussian Copula Function......Page 344
Glossary......Page 345
References......Page 347
Section 16.1. First Analysis (Nearly) Always Wrong......Page 350
Section 16.2. Why Reanalysis Is More Important Than Analysis......Page 353
Section 16.3. Case Study: Reanalysis of Old JADE Collider Data......Page 355
Section 16.5. Case Study: Finding New Planets From Old Data......Page 356
References......Page 358
Section 17.1. What Is Data Repurposing?......Page 362
Section 17.2. Dark Data, Abandoned Data, and Legacy Data......Page 364
Section 17.3. Case Study: From Postal Code to Demographic Keystone......Page 366
Section 17.4. Case Study: Scientific Inferencing From a Database of Genetic Sequences......Page 367
Section 17.6. Case Study: Inferring Climate Trends With Geologic Data......Page 368
Section 17.7. Case Study: Lunar Orbiter Image Recovery Project......Page 369
Glossary......Page 370
References......Page 371
Section 18.1. What Is Data Sharing, and Why Don’t We Do More of It?......Page 372
Section 18.2. Common Complaints......Page 373
Section 18.3. Data Security and Cryptographic Protocols......Page 380
Section 18.4. Case Study: Life on Mars......Page 386
Section 18.5. Case Study: Personal Identifiers......Page 387
Glossary......Page 389
References......Page 390
Section 19.1. Responsibility for the Accuracy and Legitimacy of Data......Page 393
Section 19.2. Rights to Create, Use, and Share the Resource......Page 396
Section 19.3. Copyright and Patent Infringements Incurred by Using Standards......Page 398
Section 19.4. Protections for Individuals......Page 400
Section 19.5. Consent......Page 402
Section 19.6. Unconsented Data......Page 407
Section 19.7. Privacy Policies......Page 409
Section 19.8. Case Study: Timely Access to Big Data......Page 410
Section 19.9. Case Study: The Havasupai Story......Page 411
Glossary......Page 413
References......Page 414
Section 20.1. How Big Data Is Perceived by the Public......Page 416
Section 20.2. Reducing Costs and Increasing Productivity With Big Data......Page 419
Section 20.3. Public Mistrust......Page 421
Section 20.4. Saving Us From Ourselves......Page 422
Section 20.5. Who Is Big Data?......Page 425
Section 20.6. Hubris and Hyperbole......Page 431
Section 20.7. Case Study: The Citizen Scientists......Page 434
Section 20.8. Case Study: 1984, by George Orwell......Page 437
Glossary......Page 438
References......Page 439
Index......Page 442