Big Data Analytics: A Guide to Data Science Practitioners Making the Transition to Big Data (Chapman & Hall/CRC Data Science)

Preface

I Setting the Scene: Analyzing Big Data

    Introduction

    1 What is Big in “Big Data”?

    2 Approaches to Analyzing Big Data

    3 The Two Domains of Big Data Analytics

        3.1 A practical big P problem

            3.1.1 Simple logistic regression (naive approach)

            3.1.2 Regularization: the lasso estimator

        3.2 A practical big N problem

            3.2.1 OLS as a point of reference

            3.2.2 The Uluru algorithm as an alternative to OLS

II Platform: Software and Computing Resources

    Introduction

    4 Software: Programming with (Big) Data

        4.1 Domains of programming with (big) data

        4.2 Measuring R performance

        4.3 Writing efficient R code

            4.3.1 Memory allocation and growing objects

            4.3.2 Vectorization in basic R functions

            4.3.3 apply-type functions and vectorization

            4.3.4 Avoiding unnecessary copying

            4.3.5 Releasing memory

            4.3.6 Beyond R

        4.4 SQL basics

            4.4.1 First steps in SQL(ite)

            4.4.2 Joins

        4.5 With a little help from my friends: GPT and R/SQL coding

        4.6 Wrapping up

    5 Hardware: Computing Resources

        5.1 Mass storage

            5.1.1 Avoiding redundancies

            5.1.2 Data compression

        5.2 Random access memory (RAM)

        5.3 Combining RAM and hard disk: Virtual memory

        5.4 CPU and parallelization

            5.4.1 Naive multi-session approach

            5.4.2 Multi-session approach with futures

            5.4.3 Multi-core and multi-node approach

        5.5 GPUs for scientific computing

            5.5.1 GPUs in R

        5.6 The road ahead: Hardware made for machine learning

        5.7 Wrapping up

        5.8 Still have insufficient computing resources?

    6 Distributed Systems

        6.1 MapReduce

        6.2 Apache Hadoop

            6.2.1 Hadoop word count example

        6.3 Apache Spark

        6.4 Spark with R

            6.4.1 Data import and summary statistics

        6.5 Spark with SQL

        6.6 Spark with R + SQL

        6.7 Wrapping up

    7 Cloud Computing

        7.1 Cloud computing basics and platforms

        7.2 Transitioning to the cloud

        7.3 Scaling up in the cloud: Virtual servers

            7.3.1 Parallelization with an EC2 instance

        7.4 Scaling up with GPUs

            7.4.1 GPUs on Google Colab

            7.4.2 RStudio and EC2 with GPUs on AWS

        7.5 Scaling out: MapReduce in the cloud

        7.6 Wrapping up

III Components of Big Data Analytics

    Introduction

    8 Data Collection and Data Storage

        8.1 Gathering and compilation of raw data

        8.2 Stack/combine raw source files

        8.3 Efficient local data storage

            8.3.1 RDBMS basics

            8.3.2 Efficient data access: Indices and joins in SQLite

        8.4 Connecting R to an RDBMS

            8.4.1 Creating a new database with RSQLite

            8.4.2 Importing data

            8.4.3 Issuing queries

        8.5 Cloud solutions for (big) data storage

            8.5.1 Easy-to-use RDBMS in the cloud: AWS RDS

        8.6 Column-based analytics databases

            8.6.1 Installation and start up

            8.6.2 First steps via Druid\'s GUI

            8.6.3 Query Druid from R

        8.7 Data warehouses

            8.7.1 Data warehouse for analytics: Google BigQuery example

        8.8 Data lakes and simple storage service

            8.8.1 AWS S3 with R: First steps

            8.8.2 Uploading data to S3

            8.8.3 More than just simple storage: S3 + Amazon Athena

        8.9 Wrapping up

    9 Big Data Cleaning and Transformation

        9.1 Out-of-memory strategies and lazy evaluation: Practical basics

            9.1.1 Chunking data with the ff package

            9.1.2 Memory mapping with bigmemory

            9.1.3 Connecting to Apache Arrow

        9.2 Big Data preparation tutorial with ff

            9.2.1 Set up

            9.2.2 Data import

            9.2.3 Inspect imported files

            9.2.4 Data cleaning and transformation

            9.2.5 Inspect difference in in-memory operation

            9.2.6 Subsetting

            9.2.7 Save/load/export ff files

        9.3 Big Data preparation tutorial with arrow

        9.4 Wrapping up

    10 Descriptive Statistics and Aggregation

        10.1 Data aggregation: The ‘split-apply-combine’ strategy

        10.2 Data aggregation with chunked data files

        10.3 High-speed in-memory data aggregation with arrow

        10.4 High-speed in-memory data aggregation with data.table

        10.5 Wrapping up

    11 (Big) Data Visualization

        11.1 Challenges of Big Data visualization

        11.2 Data exploration with ggplot2

        11.3 Visualizing time and space

            11.3.1 Preparations

            11.3.2 Pick-up and drop-off locations

        11.4 Wrapping up

IV Application: Topics in Big Data Econometrics

    Introduction

    12 Bottlenecks in Everyday Data Analytics Tasks

        12.1 Case study: Efficient fixed effects estimation

        12.2 Case study: Loops, memory, and vectorization

            12.2.1 Naïve approach (ignorant of R)

            12.2.2 Improvement 1: Pre-allocation of memory

            12.2.3 Improvement 2: Exploit vectorization

        12.3 Case study: Bootstrapping and parallel processing

            12.3.1 Parallelization with an EC2 instance

    13 Econometrics with GPUs

        13.1 OLS on GPUs

        13.2 A word of caution

        13.3 Higher-level interfaces for basic econometrics with GPUs

        13.4 TensorFlow/Keras example: Predict housing prices

            13.4.1 Data preparation

            13.4.2 Model specification

            13.4.3 Training and prediction

        13.5 Wrapping up

    14 Regression Analysis and Categorization with Spark and R

        14.1 Simple linear regression analysis

        14.2 Machine learning for classification

        14.3 Building machine learning pipelines with R and Spark

            14.3.1 Set up and data import

            14.3.2 Building the pipeline

        14.4 Wrapping up

    15 Large-scale Text Analysis with sparklyr

        15.1 Getting started: Import, pre-processing, and word count

        15.2 Tutorial: political slant

            15.2.1 Data download and import

            15.2.2 Cleaning speeches data

            15.2.3 Create a bigrams count per party

            15.2.4 Find “partisan” phrases

            15.2.5 Results: Most partisan phrases by congress

        15.3 Natural Language Processing at Scale

            15.3.1 Preparatory steps

            15.3.2 Sentiment annotation

        15.4 Aggregation and visualization

        15.5 sparklyr and lazy evaluation

V Appendices

    Appendix A: GitHub

    Appendix B: R Basics

    Appendix C: Install Hadoop

VI References and Index

    Bibliography

    Index