Data Science: A First Introduction with Python

Preface

Foreword

Acknowledgments

About the authors

1 Python and Pandas

    1.1 Overview

    1.2 Chapter learning objectives

    1.3 Canadian languages data set

    1.4 Asking a question

    1.5 Loading a tabular data set

    1.6 Naming things in Python

    1.7 Creating subsets of data frames with [ ] & loc[]

        1.7.1 Using [ ] to filter rows

        1.7.2 Using [ ] to select columns

        1.7.3 Using loc[] to filter rows and select columns

    1.8 Using sort_values and head to select rows by ordered values

    1.9 Adding and modifying columns

    1.10 Combining steps with chaining and multiline expressions

    1.11 Exploring data with visualizations

        1.11.1 Using altair to create a bar plot

        1.11.2 Formatting altair charts

        1.11.3 Putting it all together

    1.12 Accessing documentation

    1.13 Exercises

2 Reading in data locally and from the web

    2.1 Overview

    2.2 Chapter learning objectives

    2.3 Absolute and relative file paths

    2.4 Reading tabular data from a plain text file into Python

        2.4.1 read_csv to read in commaseparated values files

        2.4.2 Skipping rows when reading in data

        2.4.3 Using the sep argument for different separators

        2.4.4 Using the header argument to handle missing column names

        2.4.5 Reading tabular data directly from a URL

        2.4.6 Previewing a data file before reading it into Python

    2.5 Reading tabular data from a Microsoft Excel file

    2.6 Reading data from a database

        2.6.1 Reading data from a SQLite database

        2.6.2 Reading data from a PostgreSQL database

        2.6.3 Why should we bother with databases at all?

    2.7 Writing data from Python to a .csv file

    2.8 Obtaining data from the web

        2.8.1 Web scraping

        2.8.2 Using an API

    2.9 Exercises

    2.10 Additional resources

3 Cleaning and wrangling data

    3.1 Overview

    3.2 Chapter learning objectives

    3.3 Data frames and series

        3.3.1 What is a data frame?

        3.3.2 What is a series?

        3.3.3 What does this have to do with data frames?

        3.3.4 Data structures in Python

    3.4 Tidy data

        3.4.1 Tidying up: going from wide to long using melt

        3.4.2 Tidying up: going from long to wide using pivot

        3.4.3 Tidying up: using str.split to deal with multiple separators

    3.5 Using [ ] to extract rows or columns

        3.5.1 Extracting columns by name

        3.5.2 Extracting rows that have a certain value with ==

        3.5.3 Extracting rows that do not have a certain value with !=

        3.5.4 Extracting rows satisfying multiple conditions using &

        3.5.5 Extracting rows satisfying at least one condition using |

        3.5.6 Extracting rows with values in a list using isin

        3.5.7 Extracting rows above or below a threshold using > and <

        3.5.8 Extracting rows using query

    3.6 Using loc[] to filter rows and select columns

    3.7 Using iloc[] to extract rows and columns by position

    3.8 Aggregating data

        3.8.1 Calculating summary statistics on individual columns

        3.8.2 Calculating summary statistics on data frames

    3.9 Performing operations on groups of rows using groupby

    3.10 Apply functions across multiple columns

    3.11 Modifying and adding columns

    3.12 Using merge to combine data frames

    3.13 Summary

    3.14 Exercises

    3.15 Additional resources

4 Effective data visualization

    4.1 Overview

    4.2 Chapter learning objectives

    4.3 Choosing the visualization

    4.4 Refining the visualization

    4.5 Creating visualizations with altair

        4.5.1 Scatter plots and line plots: the Mauna Loa CO2 data set

        4.5.2 Scatter plots: the Old Faithful eruption time data set

        4.5.3 Axis transformation and colored scatter plots: the Canadian languages data set

        4.5.4 Bar plots: the island landmass data set

        4.5.5 Histograms: the Michelson speed of light data set

    4.6 Explaining the visualization

    4.7 Saving the visualization

    4.8 Exercises

    4.9 Additional resources

5 Classification I: training & predicting

    5.1 Overview

    5.2 Chapter learning objectives

    5.3 The classification problem

    5.4 Exploring a data set

        5.4.1 Loading the cancer data

        5.4.2 Describing the variables in the cancer data set

        5.4.3 Exploring the cancer data

    5.5 Classification with Knearest neighbors

        5.5.1 Distance between points

        5.5.2 More than two explanatory variables

        5.5.3 Summary of Knearest neighbors algorithm

    5.6 Knearest neighbors with scikitlearn

    5.7 Data preprocessing with scikitlearn

        5.7.1 Centering and scaling

        5.7.2 Balancing

        5.7.3 Missing data

    5.8 Putting it together in a Pipeline

    5.9 Exercises

6 Classification II: evaluation & tuning

    6.1 Overview

    6.2 Chapter learning objectives

    6.3 Evaluating performance

    6.4 Randomness and seeds

    6.5 Evaluating performance with scikitlearn

        6.5.1 Create the train / test split

        6.5.2 Preprocess the data

        6.5.3 Train the classifier

        6.5.4 Predict the labels in the test set

        6.5.5 Evaluate performance

        6.5.6 Critically analyze performance

    6.6 Tuning the classifier

        6.6.1 Crossvalidation

        6.6.2 Parameter value selection

        6.6.3 Under/Overfitting

        6.6.4 Evaluating on the test set

    6.7 Summary

    6.8 Predictor variable selection

        6.8.1 The effect of irrelevant predictors

        6.8.2 Finding a good subset of predictors

        6.8.3 Forward selection in Python

    6.9 Exercises

    6.10 Additional resources

7 Regression I: Knearest neighbors

    7.1 Overview

    7.2 Chapter learning objectives

    7.3 The regression problem

    7.4 Exploring a data set

    7.5 Knearest neighbors regression

    7.6 Training, evaluating, and tuning the model

    7.7 Underfitting and overfitting

    7.8 Evaluating on the test set

    7.9 Multivariable KNN regression

    7.10 Strengths and limitations of KNN regression

    7.11 Exercises

8 Regression II: linear regression

    8.1 Overview

    8.2 Chapter learning objectives

    8.3 Simple linear regression

    8.4 Linear regression in Python

    8.5 Comparing simple linear and KNN regression

    8.6 Multivariable linear regression

    8.7 Multicollinearity and outliers

        8.7.1 Outliers

        8.7.2 Multicollinearity

    8.8 Designing new predictors

    8.9 The other sides of regression

    8.10 Exercises

    8.11 Additional resources

9 Clustering

    9.1 Overview

    9.2 Chapter learning objectives

    9.3 Clustering

    9.4 An illustrative example

    9.5 Kmeans

        9.5.1 Measuring cluster quality

        9.5.2 The clustering algorithm

        9.5.3 Random restarts

        9.5.4 Choosing K

    9.6 Kmeans in Python

    9.7 Exercises

    9.8 Additional resources

10 Statistical inference

    10.1 Overview

    10.2 Chapter learning objectives

    10.3 Why do we need sampling?

    10.4 Sampling distributions

        10.4.1 Sampling distributions for proportions

        10.4.2 Sampling distributions for means

        10.4.3 Summary

    10.5 Bootstrapping

        10.5.1 Overview

        10.5.2 Bootstrapping in Python

        10.5.3 Using the bootstrap to calculate a plausible range

    10.6 Exercises

    10.7 Additional resources

11 Combining code and text with Jupyter

    11.1 Overview

    11.2 Chapter learning objectives

    11.3 Jupyter

        11.3.1 Accessing Jupyter

    11.4 Code cells

        11.4.1 Executing code cells

        11.4.2 The Kernel

        11.4.3 Creating new code cells

    11.5 Markdown cells

        11.5.1 Editing Markdown cells

        11.5.2 Creating new Markdown cells

    11.6 Saving your work

    11.7 Best practices for running a notebook

        11.7.1 Best practices for executing code cells

        11.7.2 Best practices for including Python packages in notebooks

        11.7.3 Summary of best practices for running a notebook

    11.8 Exploring data files

    11.9 Exporting to a different file format

        11.9.1 Exporting to HTML

        11.9.2 Exporting to PDF

    11.10 Creating a new Jupyter notebook

    11.11 Additional resources

12 Collaboration with version control

    12.1 Overview

    12.2 Chapter learning objectives

    12.3 What is version control, and why should I use it?

    12.4 Version control repositories

    12.5 Version control workflows

        12.5.1 Committing changes to a local repository

        12.5.2 Pushing changes to a remote repository

        12.5.3 Pulling changes from a remote repository

    12.6 Working with remote repositories using GitHub

        12.6.1 Creating a remote repository on GitHub

        12.6.2 Editing files on GitHub with the pen tool

        12.6.3 Creating files on GitHub with the “Add file” menu

    12.7 Working with local repositories using Jupyter

        12.7.1 Generating a GitHub personal access token

        12.7.2 Cloning a repository using Jupyter

        12.7.3 Specifying files to commit

        12.7.4 Making the commit

        12.7.5 Pushing the commits to GitHub

    12.8 Collaboration

        12.8.1 Giving collaborators access to your project

        12.8.2 Pulling changes from GitHub using Jupyter

        12.8.3 Handling merge conflicts

        12.8.4 Communicating using GitHub issues

    12.9 Exercises

    12.10 Additional resources

13 Setting up your computer

    13.1 Overview

    13.2 Chapter learning objectives

    13.3 Obtaining the worksheets for this book

    13.4 Working with Docker

        13.4.1 Windows

        13.4.2 MacOS

        13.4.3 Ubuntu

    13.5 Working with JupyterLab Desktop

        13.5.1 Windows

        13.5.2 MacOS

        13.5.3 Ubuntu

Bibliography

Index