Python for data analysis : Data wrangling with Pandas, Numpy, and IPython

Table of Contents......Page 5
 Conventions Used in This Book......Page 13
 Safari® Books Online......Page 14
 How to Contact Us......Page 15
 What Is This Book About?......Page 17
  Solving the “Two-Language” Problem......Page 18
 Essential Python Libraries......Page 19
  pandas......Page 20
  IPython......Page 21
 Installation and Setup......Page 22
  Windows......Page 23
  Apple OS X......Page 25
  GNU/Linux......Page 26
  Integrated Development Environments (IDEs)......Page 27
 Navigating This Book......Page 28
  Jargon......Page 29
 Acknowledgements......Page 30
Chapter 2. Introductory Examples......Page 33
 1.usa.gov data from bit.ly......Page 34
  Counting Time Zones in Pure Python......Page 35
  Counting Time Zones with pandas......Page 37
 MovieLens 1M Data Set......Page 42
  Measuring rating disagreement......Page 46
 US Baby Names 1880-2010......Page 48
  Analyzing Naming Trends......Page 52
   Measuring the increase in naming diversity......Page 53
   The “Last letter” Revolution......Page 56
   Boy names that became girl names (and vice versa)......Page 58
 Conclusions and The Path Ahead......Page 59
Chapter 3. IPython: An Interactive Computing and Development Environment......Page 61
 IPython Basics......Page 62
  Tab Completion......Page 63
  Introspection......Page 64
  The %run Command......Page 65
  Executing Code from the Clipboard......Page 66
  Keyboard Shortcuts......Page 68
  Exceptions and Tracebacks......Page 69
  Magic Commands......Page 70
  Qt-based Rich GUI Console......Page 71
  Matplotlib Integration and Pylab Mode......Page 72
  Input and Output Variables......Page 74
  Logging the Input and Output......Page 75
  Shell Commands and Aliases......Page 76
  Interactive Debugger......Page 78
   Other ways to make use of the debugger......Page 81
  Timing Code: %time and %timeit......Page 83
  Basic Profiling: %prun and %run -p......Page 84
  Profiling a Function Line-by-Line......Page 86
 Tips for Productive Code Development Using IPython......Page 88
  Code Design Tips......Page 90
   Overcome a fear of longer files......Page 91
  Making Your Own Classes IPython-friendly......Page 92
  Profiles and Configuration......Page 93
 Credits......Page 94
Chapter 4. NumPy Basics: Arrays and Vectorized Computation......Page 95
 The NumPy ndarray: A Multidimensional Array Object......Page 96
  Creating ndarrays......Page 97
  Data Types for ndarrays......Page 99
  Operations between Arrays and Scalars......Page 101
  Basic Indexing and Slicing......Page 102
   Indexing with slices......Page 104
  Boolean Indexing......Page 105
  Fancy Indexing......Page 108
  Transposing Arrays and Swapping Axes......Page 109
 Universal Functions: Fast Element-wise Array Functions......Page 111
 Data Processing Using Arrays......Page 113
  Expressing Conditional Logic as Array Operations......Page 114
  Mathematical and Statistical Methods......Page 116
  Sorting......Page 117
  Unique and Other Set Logic......Page 118
  Storing Arrays on Disk in Binary Format......Page 119
  Saving and Loading Text Files......Page 120
 Linear Algebra......Page 121
 Random Number Generation......Page 122
 Example: Random Walks......Page 124
  Simulating Many Random Walks at Once......Page 125
Chapter 5. Getting Started with pandas......Page 127
  Series......Page 128
  DataFrame......Page 131
  Index Objects......Page 136
  Reindexing......Page 138
  Indexing, selection, and filtering......Page 141
  Arithmetic and data alignment......Page 144
   Arithmetic methods with fill values......Page 145
   Operations between DataFrame and Series......Page 146
  Function application and mapping......Page 148
  Sorting and ranking......Page 149
  Axis indexes with duplicate values......Page 152
 Summarizing and Computing Descriptive Statistics......Page 153
  Correlation and Covariance......Page 155
  Unique Values, Value Counts, and Membership......Page 157
 Handling Missing Data......Page 158
  Filtering Out Missing Data......Page 159
  Filling in Missing Data......Page 161
 Hierarchical Indexing......Page 163
  Reordering and Sorting Levels......Page 165
  Using a DataFrame’s Columns......Page 166
  Integer Indexing......Page 167
  Panel Data......Page 168
 Reading and Writing Data in Text Format......Page 171
  Reading Text Files in Pieces......Page 176
  Writing Data Out to Text Format......Page 178
  Manually Working with Delimited Formats......Page 179
  JSON Data......Page 181
  XML and HTML: Web Scraping......Page 182
   Parsing XML with lxml.objectify......Page 185
  Using HDF5 Format......Page 187
  Reading Microsoft Excel Files......Page 188
 Interacting with HTML and Web APIs......Page 189
 Interacting with Databases......Page 190
  Storing and Loading Data in MongoDB......Page 192
 Combining and Merging Data Sets......Page 193
  Database-style DataFrame Merges......Page 194
  Merging on Index......Page 198
  Concatenating Along an Axis......Page 201
  Combining Data with Overlap......Page 204
 Reshaping and Pivoting......Page 205
  Reshaping with Hierarchical Indexing......Page 206
  Pivoting “long” to “wide” Format......Page 208
  Removing Duplicates......Page 210
  Transforming Data Using a Function or Mapping......Page 211
  Replacing Values......Page 212
  Renaming Axis Indexes......Page 213
  Discretization and Binning......Page 215
  Detecting and Filtering Outliers......Page 217
  Permutation and Random Sampling......Page 218
  Computing Indicator/Dummy Variables......Page 219
 String Manipulation......Page 221
  String Object Methods......Page 222
  Regular expressions......Page 223
  Vectorized string functions in pandas......Page 226
 Example: USDA Food Database......Page 228
 A Brief matplotlib API Primer......Page 235
  Figures and Subplots......Page 236
   Adjusting the spacing around subplots......Page 239
  Colors, Markers, and Line Styles......Page 240
  Ticks, Labels, and Legends......Page 241
   Setting the title, axis labels, ticks, and ticklabels......Page 242
  Annotations and Drawing on a Subplot......Page 244
  matplotlib Configuration......Page 247
  Line Plots......Page 248
  Bar Plots......Page 251
  Histograms and Density Plots......Page 254
  Scatter Plots......Page 255
 Plotting Maps: Visualizing Haiti Earthquake Crisis Data......Page 257
 Python Visualization Tool Ecosystem......Page 263
  Chaco......Page 264
  The Future of Visualization Tools?......Page 265
Chapter 9. Data Aggregation and Group Operations......Page 267
 GroupBy Mechanics......Page 268
  Iterating Over Groups......Page 271
  Selecting a Column or Subset of Columns......Page 272
  Grouping with Dicts and Series......Page 273
  Grouping with Functions......Page 274
 Data Aggregation......Page 275
  Column-wise and Multiple Function Application......Page 278
 Group-wise Operations and Transformations......Page 280
  Apply: General split-apply-combine......Page 282
  Quantile and Bucket Analysis......Page 284
  Example: Filling Missing Values with Group-specific Values......Page 286
  Example: Random Sampling and Permutation......Page 287
  Example: Group Weighted Average and Correlation......Page 289
  Example: Group-wise Linear Regression......Page 290
 Pivot Tables and Cross-Tabulation......Page 291
  Cross-Tabulations: Crosstab......Page 293
 Example: 2012 Federal Election Commission Database......Page 294
  Donation Statistics by Occupation and Employer......Page 296
  Bucketing Donation Amounts......Page 299
  Donation Statistics by State......Page 301
Chapter 10. Time Series......Page 305
 Date and Time Data Types and Tools......Page 306
  Converting between string and datetime......Page 307
 Time Series Basics......Page 309
  Indexing, Selection, Subsetting......Page 310
  Time Series with Duplicate Indices......Page 312
 Date Ranges, Frequencies, and Shifting......Page 313
  Generating Date Ranges......Page 314
  Frequencies and Date Offsets......Page 315
  Shifting (Leading and Lagging) Data......Page 317
   Shifting dates with offsets......Page 318
 Time Zone Handling......Page 319
  Localization and Conversion......Page 320
  Operations with Time Zone−aware Timestamp Objects......Page 321
  Operations between Different Time Zones......Page 322
 Periods and Period Arithmetic......Page 323
  Period Frequency Conversion......Page 324
  Quarterly Period Frequencies......Page 325
  Converting Timestamps to Periods (and Back)......Page 327
 Resampling and Frequency Conversion......Page 328
  Downsampling......Page 330
  Upsampling and Interpolation......Page 332
  Resampling with Periods......Page 334
 Time Series Plotting......Page 335
 Moving Window Functions......Page 336
  Binary Moving Window Functions......Page 340
  User-Defined Moving Window Functions......Page 342
 Performance and Memory Usage Notes......Page 343
 Data Munging Topics......Page 345
  Time Series and Cross-Section Alignment......Page 346
  Operations with Time Series of Different Frequencies......Page 348
   Using periods instead of timestamps......Page 349
  Time of Day and “as of” Data Selection......Page 350
  Splicing Together Data Sources......Page 352
  Return Indexes and Cumulative Returns......Page 354
 Group Transforms and Analysis......Page 356
  Group Factor Exposures......Page 358
  Decile and Quartile Analysis......Page 359
  Signal Frontier Analysis......Page 361
  Future Contract Rolling......Page 363
  Rolling Correlation and Linear Regression......Page 366
 ndarray Object Internals......Page 369
  NumPy dtype Hierarchy......Page 370
  Reshaping Arrays......Page 371
  C versus Fortran Order......Page 372
  Concatenating and Splitting Arrays......Page 373
   Stacking helpers: r_ and c_......Page 375
  Repeating Elements: Tile and Repeat......Page 376
  Fancy Indexing Equivalents: Take and Put......Page 377
 Broadcasting......Page 378
  Broadcasting Over Other Axes......Page 380
 Advanced ufunc Usage......Page 383
  ufunc Instance Methods......Page 384
 Structured and Record Arrays......Page 386
  Nested dtypes and Multidimensional Fields......Page 387
  Structured Array Manipulations: numpy.lib.recfunctions......Page 388
 More About Sorting......Page 389
  Indirect Sorts: argsort and lexsort......Page 390
  Alternate Sort Algorithms......Page 391
  numpy.searchsorted: Finding elements in a Sorted Array......Page 392
 NumPy Matrix Class......Page 393
  Memory-mapped Files......Page 395
 Performance Tips......Page 396
  The Importance of Contiguous Memory......Page 397
  Other Speed Options: Cython, f2py, C......Page 398
Appendix. Python Language Essentials......Page 401
 The Python Interpreter......Page 402
   Indentation, not braces......Page 403
   Comments......Page 404
   Variables and pass-by-reference......Page 405
   Dynamic references, strong types......Page 406
   Attributes and methods......Page 407
   Imports......Page 408
   Binary operators and comparisons......Page 409
   Mutable and immutable objects......Page 410
   Numeric types......Page 411
   Strings......Page 412
   Booleans......Page 414
   Dates and times......Page 415
   if, elif, and else......Page 416
   for loops......Page 417
   Exception handling......Page 418
   range and xrange......Page 420
 Data Structures and Sequences......Page 421
  Tuple......Page 422
   Tuple methods......Page 423
   Adding and removing elements......Page 424
   Sorting......Page 425
   Slicing......Page 426
  Built-in Sequence Functions......Page 427
   zip......Page 428
  Dict......Page 429
   Default values......Page 431
  Set......Page 432
  List, Set, and Dict Comprehensions......Page 434
   Nested list comprehensions......Page 435
  Namespaces, Scope, and Local Functions......Page 436
  Functions Are Objects......Page 438
  Anonymous (lambda) Functions......Page 440
  Closures: Functions that Return Functions......Page 441
  Extended Call Syntax with *args, **kwargs......Page 442
  Generators......Page 443
   itertools module......Page 445
 Files and the operating system......Page 446
Index......Page 449