Learning predictive analytics with Python: gain practical insights into predictive modelling by implementing predictive analytics algorithms on public datasets with Python

Learning Predictive Analytics with Python......Page 15
Credits......Page 17
Foreword......Page 19
About the Author......Page 22
Acknowledgments......Page 24
About the Reviewer......Page 26
www.PacktPub.com......Page 28
Support files, eBooks, discount offers, and more......Page 30
Why subscribe?......Page 31
Free access for Packt account holders......Page 32
Preface......Page 33
What this book covers......Page 35
What you need for this book......Page 37
Who this book is for......Page 39
Conventions......Page 41
Reader feedback......Page 43
Customer support......Page 45
Downloading the example code......Page 47
Downloading the color images of this book......Page 48
Errata......Page 49
Piracy......Page 50
Questions......Page 51
1. Getting Started with Predictive Modelling......Page 52
Introducing predictive modelling......Page 54
Statistical tools......Page 56
Mathematical function......Page 57
Business context......Page 58
Knowledge matrix for predictive modelling......Page 59
Task matrix for predictive modelling......Page 60
Applications and examples of predictive modelling......Page 61
How is it done?......Page 63
How is it done?......Page 64
How is it done?......Page 65
How is it done?......Page 66
How was it done?......Page 67
Python and its packages – download and installation......Page 68
Anaconda......Page 70
Standalone Python......Page 71
Installing pip......Page 72
Installing Python packages with pip......Page 74
Python and its packages for predictive modelling......Page 76
IDEs for Python......Page 79
Summary......Page 83
2. Data Cleaning......Page 85
Reading the data – variations and examples......Page 87
Data frames......Page 88
Delimiters......Page 89
Various methods of importing data in Python......Page 90
Case 1 – reading a dataset using the read_csv method......Page 92
The read_csv method......Page 93
Use cases of the read_csv method......Page 96
Passing the directory address and filename as variables......Page 98
Reading a .txt dataset with a comma delimiter......Page 99
Specifying the column names of a dataset from a list......Page 100
Case 2 – reading a dataset using the open method of Python......Page 102
Reading a dataset line by line......Page 104
Changing the delimiter of a dataset......Page 106
Case 3 – reading data from a URL......Page 108
Case 4 – miscellaneous cases......Page 110
Reading from an .xls or .xlsx file......Page 112
Writing to a CSV or Excel file......Page 113
Basics – summary, dimensions, and structure......Page 114
Handling missing values......Page 117
Checking for missing values......Page 119
How missing values are generated and propagated......Page 120
Imputation......Page 121
Creating dummy variables......Page 125
Visualizing a dataset by basic plotting......Page 127
Scatter plots......Page 129
Histograms......Page 131
Boxplots......Page 132
Summary......Page 134
3. Data Wrangling......Page 136
Subsetting a dataset......Page 139
Selecting columns......Page 140
Selecting rows......Page 143
Selecting a combination of rows and columns......Page 145
Creating new columns......Page 147
Generating random numbers and their usage......Page 148
Various methods for generating random numbers......Page 150
Seeding a random number......Page 153
Cumulative density function......Page 154
Uniform distribution......Page 155
Normal distribution......Page 157
Geometry and mathematics behind the calculation of pi......Page 160
Generating a dummy data frame......Page 163
Grouping the data – aggregation, filtering, and transformation......Page 165
Aggregation......Page 170
Filtering......Page 173
Transformation......Page 174
Miscellaneous operations......Page 175
Random sampling – splitting a dataset in training and testing datasets......Page 177
Method 1 – using the Customer Churn Model......Page 180
Method 2 – using sklearn......Page 181
Method 3 – using the shuffle function......Page 182
Concatenating and appending data......Page 183
Merging/joining datasets......Page 191
Inner Join......Page 197
Left Join......Page 198
Right Join......Page 199
An example of the Inner Join......Page 201
An example of the Left Join......Page 202
An example of the Right Join......Page 203
Summary of Joins in terms of their length......Page 204
Summary......Page 205
4. Statistical Concepts for Predictive Modelling......Page 207
Random sampling and the central limit theorem......Page 209
Hypothesis testing......Page 210
Null versus alternate hypothesis......Page 212
Z-statistic and t-statistic......Page 213
Confidence intervals, significance levels, and p-values......Page 215
Different kinds of hypothesis test......Page 218
A step-by-step guide to do a hypothesis test......Page 220
An example of a hypothesis test......Page 221
Chi-square tests......Page 223
Correlation......Page 228
Summary......Page 234
5. Linear Regression with Python......Page 236
Understanding the maths behind linear regression......Page 239
Fitting a linear regression model and checking its efficacy......Page 242
Finding the optimum value of variable coefficients......Page 246
Making sense of result parameters......Page 249
p-values......Page 251
F-statistics......Page 252
Residual Standard Error......Page 253
Implementing linear regression with Python......Page 254
Linear regression using the statsmodel library......Page 256
Multiple linear regression......Page 259
Variance Inflation Factor......Page 265
Model validation......Page 267
Training and testing data split......Page 269
Summary of models......Page 271
Linear regression with scikit-learn......Page 272
Feature selection with scikit-learn......Page 273
Handling other issues in linear regression......Page 275
Handling categorical variables......Page 277
Transforming a variable to fit non-linear relations......Page 284
Handling outliers......Page 290
Other considerations and assumptions for linear regression......Page 295
Summary......Page 298
6. Logistic Regression with Python......Page 301
Linear regression versus logistic regression......Page 303
Understanding the math behind logistic regression......Page 304
Contingency tables......Page 306
Conditional probability......Page 308
Odds ratio......Page 309
Moving on to logistic regression from linear regression......Page 311
Estimation using the Maximum Likelihood Method......Page 314
Likelihood function:......Page 315
Log likelihood function:......Page 316
Building the logistic regression model from scratch......Page 317
Likelihood Ratio Test statistic......Page 320
Chi-square test......Page 321
Implementing logistic regression with Python......Page 322
Processing the data......Page 325
Data exploration......Page 327
Data visualization......Page 329
Creating dummy variables for categorical variables......Page 334
Feature selection......Page 336
Implementing the model......Page 337
Model validation and evaluation......Page 339
Cross validation......Page 342
Model validation......Page 344
The ROC curve......Page 346
Confusion matrix......Page 348
Summary......Page 353
7. Clustering with Python......Page 355
Introduction to clustering – what, why, and how?......Page 357
What is clustering?......Page 358
How is clustering used?......Page 359
Why do we do clustering?......Page 361
Mathematics behind clustering......Page 362
Euclidean distance......Page 364
The distance matrix......Page 365
Normalizing the distances......Page 367
Centroid linkage......Page 369
Ward's method......Page 370
Hierarchical clustering......Page 371
K-means clustering......Page 375
Implementing clustering using Python......Page 378
Importing and exploring the dataset......Page 380
Normalizing the values in the dataset......Page 382
Hierarchical clustering using scikit-learn......Page 383
K-Means clustering using scikit-learn......Page 385
Interpreting the cluster......Page 387
Fine-tuning the clustering......Page 389
The elbow method......Page 391
Silhouette Coefficient......Page 393
Summary......Page 395
8. Trees and Random Forests with Python......Page 397
Introducing decision trees......Page 399
A decision tree......Page 400
Understanding the mathematics behind decision trees......Page 403
Homogeneity......Page 405
Entropy......Page 407
Information gain......Page 409
ID3 algorithm to create a decision tree......Page 411
Gini index......Page 412
Reduction in Variance......Page 413
Pruning a tree......Page 414
Handling a continuous numerical variable......Page 415
Handling a missing value of an attribute......Page 416
Implementing a decision tree with scikit-learn......Page 418
Visualizing the tree......Page 422
Cross-validating and pruning the decision tree......Page 425
Understanding and implementing regression trees......Page 427
Regression tree algorithm......Page 429
Implementing a regression tree using Python......Page 431
Understanding and implementing random forests......Page 434
The random forest algorithm......Page 436
Implementing a random forest using Python......Page 437
Why do random forests work?......Page 439
Important parameters for random forests......Page 440
Summary......Page 441
9. Best Practices for Predictive Modelling......Page 443
Best practices for coding......Page 445
Commenting the codes......Page 446
Example 2......Page 448
Example 3......Page 449
Avoid hard-coding of variables as much as possible......Page 450
Version control......Page 451
Using standard libraries, methods, and formulas......Page 452
Best practices for data handling......Page 453
Best practices for algorithms......Page 455
Best practices for statistics......Page 457
Best practices for business contexts......Page 459
Summary......Page 461
A. A List of Links......Page 463
Index......Page 466