Python for Data Science, 2nd Edition

Content: Introduction 1About This Book 1Foolish Assumptions 3Icons Used in This Book 4Beyond the Book 4Where to Go from Here 5Part 1: Getting Started With Data Science and Python 7Chapter 1: Discovering the Match between Data Science and Python 9Defining the Sexiest Job of the 21st Century 11Considering the emergence of data science 12Outlining the core competencies of a data scientist 12Linking data science, big data, and AI 13Understanding the role of programming 14Creating the Data Science Pipeline 14Preparing the data 15Performing exploratory data analysis 15Learning from data 15Visualizing 15Obtaining insights and data products 16Understanding Python\'s Role in Data Science 16Considering the shifting profile of data scientists 16Working with a multipurpose, simple, and efficient language 17Learning to Use Python Fast 18Loading data 19Training a model 19Viewing a result 19Chapter 2: Introducing Python\'s Capabilities and Wonders 21Why Python? 22Grasping Python\'s Core Philosophy 23Contributing to data science 23Discovering present and future development goals 24Working with Python 25Getting a taste of the language 25Understanding the need for indentation 26Working at the command line or in the IDE 27Performing Rapid Prototyping and Experimentation 31Considering Speed of Execution 32Visualizing Power 33Using the Python Ecosystem for Data Science 35Accessing scientific tools using SciPy 35Performing fundamental scientific computing using NumPy 36Performing data analysis using pandas 36Implementing machine learning using Scikit-learn 36Going for deep learning with Keras and TensorFlow 37Plotting the data using matplotlib 38Creating graphs with NetworkX 38Parsing HTML documents using Beautiful Soup 38Chapter 3: Setting Up Python for Data Science 39Considering the Off-the-Shelf Cross-Platform Scientific Distributions 40Getting Continuum Analytics Anaconda 40Getting Enthought Canopy Express 41Getting WinPython 42Installing Anaconda on Windows 42Installing Anaconda on Linux 46Installing Anaconda on Mac OS X 47Downloading the Datasets and Example Code 48Using Jupyter Notebook 49Defining the code repository 50Understanding the datasets used in this book 57Chapter 4: Working with Google Colab 59Defining Google Colab 60Understanding what Google Colab does 60Considering the online coding difference 61Using local runtime support 63Getting a Google Account 63Creating the account 64Signing in 64Working with Notebooks 65Creating a new notebook 65Opening existing notebooks 66Saving notebooks 68Downloading notebooks 71Performing Common Tasks 71Creating code cells 71Creating text cells 72Creating special cells 73Editing cells 74Moving cells 75Using Hardware Acceleration 75Executing the Code 76Viewing Your Notebook 76Displaying the table of contents 77Getting notebook information 77Checking code execution 78Sharing Your Notebook 79Getting Help 80Part 2: Getting Your Hands Dirty With Data 81Chapter 5: Understanding the Tools 83Using the Jupyter Console 84Interacting with screen text 84Changing the window appearance 86Getting Python help 87Getting IPython help 89Using magic functions 90Discovering objects 91Using Jupyter Notebook 93Working with styles 93Restarting the kernel 94Restoring a checkpoint 95Performing Multimedia and Graphic Integration 96Embedding plots and other images 96Loading examples from online sites 96Obtaining online graphics and multimedia 96Chapter 6: Working with Real Data 99Uploading, Streaming, and Sampling Data 100Uploading small amounts of data into memory 101Streaming large amounts of data into memory 102Generating variations on image data 103Sampling data in different ways 104Accessing Data in Structured Flat-File Form 105Reading from a text file 106Reading CSV delimited format 107Reading Excel and other Microsoft Office files 109Sending Data in Unstructured File Form 111Managing Data from Relational Databases 113Interacting with Data from NoSQL Databases 115Accessing Data from the Web 116Chapter 7: Conditioning Your Data 121Juggling between NumPy and pandas 122Knowing when to use NumPy 122Knowing when to use pandas 122Validating Your Data 124Figuring out what\'s in your data 124Removing duplicates 126Creating a data map and data plan 126Manipulating Categorical Variables 129Creating categorical variables 130Renaming levels 131Combining levels 132Dealing with Dates in Your Data 133Formatting date and time values 134Using the right time transformation 135Dealing with Missing Data 136Finding the missing data 136Encoding missingness 137Imputing missing data 138Slicing and Dicing: Filtering and Selecting Data 139Slicing rows 140Slicing columns 140Dicing 141Concatenating and Transforming 142Adding new cases and variables 142Removing data 144Sorting and shuffling 145Aggregating Data at Any Level 146Chapter 8: Shaping Data 149Working with HTML Pages 150Parsing XML and HTML 150Using XPath for data extraction 151Working with Raw Text 153Dealing with Unicode 153Stemming and removing stop words 153Introducing regular expressions 155Using the Bag of Words Model and Beyond 158Understanding the bag of words model 159Working with n-grams 161Implementing TF-IDF transformations 162Working with Graph Data 165Understanding the adjacency matrix 165Using NetworkX basics 166Chapter 9: Putting What You Know in Action 169Contextualizing Problems and Data 170Evaluating a data science problem 171Researching solutions 173Formulating a hypothesis 174Preparing your data 175Considering the Art of Feature Creation 175Defining feature creation 175Combining variables 176Understanding binning and discretization 177Using indicator variables 177Transforming distributions 178Performing Operations on Arrays 178Using vectorization 179Performing simple arithmetic on vectors and matrices 179Performing matrix vector multiplication 180Performing matrix multiplication 181Part 3: Visualizing Information 183Chapter 10: Getting a Crash Course in MatPlotLib 185Starting with a Graph 186Defining the plot 186Drawing multiple lines and plots 187Saving your work to disk 188Setting the Axis, Ticks, Grids 189Getting the axes 189Formatting the axes 190Adding grids 191Defining the Line Appearance 192Working with line styles 193Using colors 194Adding markers 195Using Labels, Annotations, and Legends 197Adding labels 198Annotating the chart 198Creating a legend 199Chapter 11: Visualizing the Data 201Choosing the Right Graph 202Showing parts of a whole with pie charts 202Creating comparisons with bar charts 203Showing distributions using histograms 205Depicting groups using boxplots 206Seeing data patterns using scatterplots 208Creating Advanced Scatterplots 209Depicting groups 209Showing correlations 211Plotting Time Series 212Representing time on axes 212Plotting trends over time 214Plotting Geographical Data 216Using an environment in Notebook 217Getting the Basemap toolkit 218Dealing with deprecated library issues 218Using Basemap to plot geographic data 220Visualizing Graphs 221Developing undirected graphs 222Developing directed graphs 224Part 4: Wrangling Data 227Chapter 12: Stretching Python\'s Capabilities 229Playing with Scikit-learn 230Understanding classes in Scikit-learn 230Defining applications for data science 231Performing the Hashing Trick 234Using hash functions 235Demonstrating the hashing trick 235Working with deterministic selection 239Considering Timing and Performance 240Benchmarking with timeit 241Working with the memory profiler 244Running in Parallel on Multiple Cores 247Performing multicore parallelism 248Demonstrating multiprocessing 248Chapter 13: Exploring Data Analysis 251The EDA Approach 252Defining Descriptive Statistics for Numeric Data 253Measuring central tendency 254Measuring variance and range 255Working with percentiles 256Defining measures of normality 257Counting for Categorical Data 259Understanding frequencies 259Creating contingency tables 261Creating Applied Visualization for EDA 261Inspecting boxplots 262Performing t-tests after boxplots 263Observing parallel coordinates 264Graphing distributions 265Plotting scatterplots 266Understanding Correlation 268Using covariance and correlation 268Using nonparametric correlation 270Considering the chi-square test for tables 271Modifying Data Distributions 272Using different statistical distributions 272Creating a Z-score standardization 273Transforming other notable distributions 273Chapter 14: Reducing Dimensionality 275Understanding SVD 276Looking for dimensionality reduction 277Using SVD to measure the invisible 279Performing Factor Analysis and PCA 280Considering the psychometric model 280Looking for hidden factors 281Using components, not factors 282Achieving dimensionality reduction 282Squeezing information with t-SNE 283Understanding Some Applications 285Recognizing faces with PCA 285Extracting topics with NMF 289Recommending movies 291Chapter 15: Clustering 295Clustering with K-means 297Understanding centroid-based algorithms 298Creating an example with image data 299Looking for optimal solutions 301Clustering big data 304Performing Hierarchical Clustering 305Using a hierarchical cluster solution 307Using a two-phase clustering solution 308Discovering New Groups with DBScan 310Chapter 16: Detecting Outliers in Data 313Considering Outlier Detection 314Finding more things that can go wrong 315Understanding anomalies and novel data 316Examining a Simple Univariate Method 317Leveraging on the Gaussian distribution 319Making assumptions and checking out 320Developing a Multivariate Approach 322Using principal component analysis 322Using cluster analysis for spotting outliers 324Automating detection with Isolation Forests 325Part 5: Learning From Data 327Chapter 17: Exploring Four Simple and Effective Algorithms 329Guessing the Number: Linear Regression 329Defining the family of linear models 330Using more variables 331Understanding limitations and problems 333Moving to Logistic Regression 334Applying logistic regression 335Considering when classes are more 336Making Things as Simple as Naive Bayes 337Finding out that Naive Bayes isn\'t so naive 339Predicting text classifications 340Learning Lazily with Nearest Neighbors 342Predicting after observing neighbors 343Choosing your k parameter wisely 344Chapter 18: Performing Cross-Validation, Selection, and Optimization 347Pondering the Problem of Fitting a Model 348Understanding bias and variance 349Defining a strategy for picking models 350Dividing between training and test sets 354Cross-Validating 356Using cross-validation on k folds 357Sampling stratifications for complex data 358Selecting Variables Like a Pro 360Selecting by univariate measures 360Using a greedy search 362Pumping Up Your Hyperparameters 363Implementing a grid search 364Trying a randomized search 368Chapter 19: Increasing Complexity with Linear and Nonlinear Tricks 371Using Nonlinear Transformations 372Doing variable transformations 372Creating interactions between variables 375Regularizing Linear Models 379Relying on Ridge regression (L2) 380Using the Lasso (L1) 381Leveraging regularization 382Combining L1 & L2: Elasticnet 382Fighting with Big Data Chunk by Chunk 383Determining when there is too much data 383Implementing Stochastic Gradient Descent 383Understanding Support Vector Machines 387Relying on a computational method 387Fixing many new parameters 390Classifying with SVC 392Going nonlinear is easy 398Performing regression with SVR 399Creating a stochastic solution with SVM 401Playing with Neural Networks 406Understanding neural networks 407Classifying and regressing with neurons 408Chapter 20: Understanding the Power of the Many 411Starting with a Plain Decision Tree 412Understanding a decision tree 412Creating trees for different purposes 415Making Machine Learning Accessible 418Working with a Random Forest classifier 420Working with a Random Forest regressor 421Optimizing a Random Forest 422Boosting Predictions 424Knowing that many weak predictors win 424Setting a gradient boosting classifier 425Running a gradient boosting regressor 426Using GBM hyperparameters 427Part 6: The Part of Tens 429Chapter 21: Ten Essential Data Resources 431Discovering the News with Subreddit 432Getting a Good Start with KDnuggets 432Locating Free Learning Resources with Quora 432Gaining Insights with Oracle\'s Data Science Blog 433Accessing the Huge List of Resources on Data Science Central 433Learning New Tricks from the Aspirational Data Scientist 434Obtaining the Most Authoritative Sources at Udacity 435Receiving Help with Advanced Topics at Conductrics 435Obtaining the Facts of Open Source Data Science from Masters 436Zeroing In on Developer Resources with Jonathan Bower 436Chapter 22: Ten Data Challenges You Should Take 437Meeting the Data Science London + Scikit-learn Challenge 438Predicting Survival on the Titanic 438Finding a Kaggle Competition that Suits Your Needs 439Honing Your Overfit Strategies 440Trudging Through the MovieLens Dataset 440Getting Rid of Spam E-mails 441Working with Handwritten Information 442Working with Pictures 443Analyzing Amazon.com Reviews 444Interacting with a Huge Graph 444Index 447