Python Data Analytics: Data Analysis and Science Using Pandas, matplotlib, and the Python Programming Language

Contents at a Glance......Page 4
Contents......Page 5
About the Author......Page 16
About the Technical Reviewer......Page 17
Acknowledgments......Page 18
  Data Analysis......Page 19
   Computer Science......Page 20
   Professional Fields of Application......Page 21
   Types of Data......Page 22
   Problem Definition......Page 23
   Data Extraction......Page 24
   Data Exploration/Visualization......Page 25
   Deployment......Page 26
  Quantitative and Qualitative Data Analysis......Page 27
  Open Data......Page 28
  Python and Data Analysis......Page 29
  Conclusions......Page 30
  Python—The Programming Language......Page 31
  Python—The Interpreter......Page 32
  Python 2 and Python 3......Page 33
   Anaconda......Page 34
   Enthought Canopy......Page 35
   Run an Entire Program Code......Page 36
  Writing Python Code......Page 37
   Import New Libraries and Functions......Page 38
   Data Structure......Page 39
   Functional Programming (Only for Python 3.4)......Page 40
   IPython Shell......Page 42
   IPython Notebook......Page 44
   The Jupyter Project......Page 45
  The IDEs for Python......Page 46
   Spyder......Page 47
   Sublime......Page 48
   Liclipse......Page 49
  SciPy......Page 50
   Pandas......Page 51
  Conclusions......Page 52
  The NumPy Installation......Page 53
  Ndarray: The Heart of the Library......Page 54
   Create an Array......Page 55
   Types of Data......Page 56
   Intrinsic Crea tion of an Array......Page 57
  Basic Operations......Page 58
   Arit hmetic Operators......Page 59
   The M atrix Product......Page 60
   Increm ent and Decrement Operators......Page 61
   Aggregat e Functions......Page 62
   Indexing......Page 63
   Slicing......Page 64
   Iterating an Array......Page 66
  Shape Manipulation......Page 68
   Joining Arrays......Page 69
   Splitting Arrays......Page 70
   Copies or Views of Objects......Page 72
   Broadcasting......Page 73
  Structured Arrays......Page 76
   Loading and Saving Data in Binary Files......Page 77
   Reading File with T abular Data......Page 78
  Conclusions......Page 79
  pandas: The Python Data Analysis Library......Page 80
   Installation from Anaconda......Page 81
   Installation on Linux......Page 82
  Test Your pandas Installation......Page 83
  Introduction to pandas Data Structures......Page 84
   Declaring a Series......Page 85
   Selecting the Internal Elements......Page 86
   Defining Series from NumPy Arrays and Other Series......Page 87
    Operations and Mathematical Functions......Page 88
    Evaluating Values......Page 89
    NaN Values......Page 90
   Operations between Series......Page 91
   Defining a DataFrame......Page 92
    Selecting Elements......Page 94
    Assigning Values......Page 95
    Filtering......Page 97
   The Index Objects......Page 98
   Index with Duplicate Labels......Page 99
   Reindexing......Page 100
   Dropping......Page 102
   Arithmetic and Data Alignment......Page 103
  Operations between Data Structures......Page 104
   Operations between DataFrame and Series......Page 105
   Functions by Element......Page 106
   Functions by Row or Column......Page 107
  Sorting and Ranking......Page 108
  Correlation and Covariance......Page 111
  “Not a Number” Data......Page 112
   Filtering Out NaN Values......Page 113
  Hierarchical Indexing and Leveling......Page 114
   Summary Statistic by Level......Page 117
  Conclusions......Page 118
  I/O API Tools......Page 119
  Reading Data in CSV or Text Files......Page 120
   Using RegExp for Parsing TXT Files......Page 122
   Reading TXT Files into Parts or Partially......Page 124
   Writing Data in CSV......Page 125
   Writing Data in HTML......Page 127
   Reading Data from an HTML File......Page 129
  Reading Data from XML......Page 130
  Reading and Writing Data on Microsoft Excel Files......Page 132
  JSON Data......Page 134
  The Format HDF5......Page 137
   Serialize a Python Object with cPickle......Page 138
   Pickling with pandas......Page 139
   Loading and Writing Data with SQLite3......Page 140
   Loading and Writing Data with PostgreSQL......Page 142
  Reading and Writing Data with a NoSQL Database: MongoDB......Page 144
  Conclusions......Page 146
  Data Preparation......Page 147
   Merging......Page 148
   Merging on Index......Page 151
  Concatenating......Page 152
   Combining......Page 155
   Pivoting with Hierarchical Indexing......Page 156
   Pivoting from “Long” to “Wide” Format......Page 157
   Removing......Page 158
   Removing Duplicates......Page 159
   Replacing Values via Mapping......Page 160
   Adding Values via Mapping......Page 161
   Rename the Indexes of the Axes......Page 162
  Discretization and Binning......Page 164
   Detecting and Filtering Outliers......Page 167
   Random Sampling......Page 168
   Built-in Methods for Manipulation of Strings......Page 169
   Regular Expressions......Page 171
  Data Aggregation......Page 172
   GroupBy......Page 173
   A Practical Example......Page 174
   Hierarchical Grouping......Page 175
   Chain of Transformations......Page 176
   Functions on Groups......Page 177
  Advanced Data Aggregation......Page 178
  Conclusions......Page 181
  The matplotlib Library......Page 182
  IPython and IPython QtConsole......Page 183
   Backend Layer......Page 185
   Artist Layer......Page 186
   pylab and pyplot......Page 187
   A Simple Interactive Chart......Page 188
   Set the Properties of the Plot......Page 192
   matplotlib and NumPy......Page 194
  Using the kwargs......Page 196
   Working with Multiple Figures and Axes......Page 197
   Adding Text......Page 199
   Adding a Grid......Page 203
   Adding a Legend......Page 204
   Saving the Code......Page 207
   Converting Your Session as an HTML File......Page 208
   Saving Your Chart Directly as an Image......Page 210
  Handling Date Values......Page 211
  Line Chart......Page 213
   Line Charts with pandas......Page 220
  Histogram......Page 221
  Bar Chart......Page 222
   Horizontal Bar Chart......Page 225
   Multiserial Bar Chart......Page 226
   Multiseries Bar Chart with pandas DataFrame......Page 228
   Multiseries Stacked Bar Charts......Page 230
   Stacked Bar Charts with pandas DataFrame......Page 232
   Other Bar Chart Representations......Page 233
  Pie Charts......Page 234
   Pie Charts with pandas DataFrame......Page 237
   Contour Plot......Page 238
   Polar Chart......Page 240
   3D Surfaces......Page 242
   Scatter Plot in 3D......Page 244
   Bar Chart 3D......Page 245
   Display Subplots within Other Subplots......Page 246
   Grids of Subplots......Page 248
  Conclusions......Page 250
   Supervised and Unsupervised Learning......Page 251
  The Iris Flower Dataset......Page 252
   The PCA Decomposition......Page 256
  K-Nearest Neighbors Classifier......Page 258
  Diabetes Dataset......Page 261
  Linear Regression: The Least Square Regression......Page 262
   Support Vector Classification ( SVC)......Page 267
   Nonlinear SVC......Page 271
   Plotting Different SVM Classifiers Using the Iris Dataset......Page 273
   Support Vector Regression (SVR)......Page 276
  Conclusions......Page 278
   The System in the Study: The Adriatic Sea and the Po Valley......Page 279
  Data Source......Page 282
  Data Analysis on IPython Notebook......Page 284
  The RoseWind......Page 298
   Calculating the Distribution of the Wind Speed Means......Page 301
  Conclusions......Page 302
  The Open Data Source for Demographics......Page 303
  The JavaScript D3 Library......Page 307
  Drawing a Clustered Bar Chart......Page 310
  The Choropleth Maps......Page 314
  The Choropleth Map of the US Population in 2014......Page 318
  Conclusions......Page 323
  Recognizing Handwritten Digits with scikit-learn......Page 324
  The Digits Dataset......Page 325
  Learning and Predicting......Page 328
  Conclusions......Page 329
  With IPython Notebook in a Python 2 Cell......Page 330
  Radicals......Page 331
  Accents......Page 332
  Political and Government Data......Page 340
  Social Data......Page 341
  Climatic Data......Page 342
  Musical Data......Page 343
Index......Page 344