Python and HDF5: Unlocking Scientific Data

Copyright......Page 6
Table of Contents......Page 7
 Conventions Used in This Book......Page 13
 Safari® Books Online......Page 14
 Acknowledgments......Page 15
Chapter 1. Introduction......Page 17
  Organizing Data and Metadata......Page 18
  Coping with Large Data Volumes......Page 19
 What Exactly Is HDF5?......Page 20
  HDF5: The File......Page 21
  HDF5: The Ecosystem......Page 22
 HDF5 Basics......Page 23
  Python 2 or Python 3?......Page 24
  Code Examples......Page 25
  NumPy......Page 26
  IPython......Page 27
  Timing and Optimization......Page 28
  HDFView......Page 30
  Command Line Tools......Page 31
 Your First HDF5 File......Page 33
  File Drivers......Page 34
  The User Block......Page 35
  Type and Shape......Page 37
  Reading and Writing......Page 38
  Saving Space with Explicit Storage Types......Page 39
  Automatic Type Conversion and Direct Reads......Page 40
  Reading with astype......Page 41
  Fill Values......Page 42
  Using Slicing Effectively......Page 43
  Start-Stop-Step Indexing......Page 45
  Multidimensional and Scalar Slicing......Page 46
  Boolean Indexing......Page 47
  Coordinate Lists......Page 48
  Automatic Broadcasting......Page 49
  Reading Directly into an Existing Array......Page 50
  A Note on Data Types......Page 51
 Resizing Datasets......Page 52
  Creating Resizable Datasets......Page 53
  Data Shuffling with resize......Page 54
  When and How to Use resize......Page 55
 Contiguous Storage......Page 57
 Chunked Storage......Page 59
  Manually Picking a Shape......Page 61
 Performance Example: Resizable Datasets......Page 62
  The Filter Pipeline......Page 64
  Compression Filters......Page 65
  SZIP Compression......Page 66
  Performance......Page 67
  SHUFFLE Filter......Page 68
  FLETCHER32 Filter......Page 69
 Third-Party Filters......Page 70
 The Root Group and Subgroups......Page 71
  Dictionary-Style Access......Page 72
  Hard Links......Page 73
  Soft Links......Page 75
  External Links......Page 77
  A Note on Object Names......Page 78
  Using get to Determine Object Types......Page 79
  Using require to Simplify Your Application......Page 80
  How Groups Are Actually Stored......Page 81
  Dictionary-Style Iteration......Page 82
  Containership Testing......Page 83
  Visit by Name......Page 84
  Multiple Links and visit......Page 85
  Canceling Iteration: A Simple Search Mechanism......Page 86
  Single-File Copying......Page 87
 Object Comparison and Hashing......Page 88
 Attribute Basics......Page 91
  Type Guessing......Page 93
  Strings and File Compatibility......Page 94
  Explicit Typing......Page 96
  Application Format on Top of HDF5......Page 98
  Analyzing the Data......Page 100
 The HDF5 Type System......Page 103
 Integers and Floats......Page 104
 Variable-Length Strings......Page 105
  The vlen String Data Type......Page 106
  Byte Versus Unicode Strings......Page 107
  Using Unicode Strings......Page 108
 Compound Types......Page 109
 Enumerated Types......Page 111
 Booleans......Page 112
 The array Type......Page 113
 Opaque Types......Page 114
 Dates and Times......Page 115
  Creating and Resolving References......Page 117
  References as “Unbreakable” Links......Page 118
  References as Data......Page 119
  Creating Region References and Reading......Page 120
  Fancy Indexing......Page 121
 Named Types......Page 122
  Linking to Named Types......Page 123
 Dimension Scales......Page 124
  Creating Dimension Scales......Page 125
  Attaching Scales to a Dataset......Page 126
 Python Parallel Basics......Page 129
 Threading......Page 130
 Multiprocessing......Page 132
 MPI and Parallel HDF5......Page 135
  A Very Quick Introduction to MPI......Page 136
  MPI-Based HDF5 Program......Page 137
  Collective Versus Independent Operations......Page 138
  Atomicity Gotchas......Page 139
 Contributing......Page 143
Index......Page 145
About the Author......Page 152