Parallel computing for data science : with examples in R, C++ and CUDA

Introduction to Parallel Processing in R  Recurring Theme: The Principle of Pretty Good Parallelism  A Note on Machines  Recurring Theme: Hedging One's Bets  Extended Example: Mutual Web Outlinks         "Why Is My Program So Slow?": Obstacles to Speed  Obstacles to Speed  Performance and Hardware Structures  Memory Basics  Network Basics  Latency and Bandwidth Thread Scheduling  How Many Processes/Threads?  Example: Mutual Outlink Problem  "Big O" Notation  Data Serialization  "Embarrassingly Parallel" Applications         Principles of Parallel Loop Scheduling  General Notions of Loop Scheduling  Chunking in Snow  A Note on Code Complexity  Example: All Possible Regressions  The partools Package  Example: All Possible Regressions, Improved Version  Introducing Another Tool: multicore  Issues with Chunk Size  Example: Parallel Distance Computation The foreach Package  Stride  Another Scheduling Approach: Random Task Permutation  Debugging snow and multicore Code         The Shared Memory Paradigm: A Gentle Introduction through R  So, What Is Actually Shared?  Clarity and Conciseness of Shared-Memory Programming  High-Level Introduction to Shared-Memory Programming: Rdsm Package  Example: Matrix Multiplication  Shared Memory Can Bring a Performance Advantage  Locks and Barriers  Example: Finding the Maximal Burst in a Time Series  Example: Transformation of an Adjacency Matrix  Example: k-Means Clustering         The Shared Memory Paradigm: C Level  OpenMP  Example: Finding the Maximal Burst in a Time Series  OpenMP Loop Scheduling Options  Example: Transformation an Adjacency Matrix  Example: Transforming an Adjacency Matrix, R-Callable Code  Speedup in C  Run Time vs. Development Time  Further Cache/Virtual Memory Issues  Reduction Operations in OpenMP  Debugging Intel Thread Building Blocks (TBB)  Lockfree Synchronization         The Shared Memory Paradigm: GPUs  Overview  Another Note on Code Complexity  Goal of This Chapter  Introduction to NVIDIA GPUs and CUDA  Example: Mutual Inlinks Problem  Synchronization on GPUs  R and GPUs  The Intel Xeon Phi Chip         Thrust and Rth  Hedging One's Bets  Thrust Overview  Rth  Skipping the C++  Example: Finding Quantiles Introduction to Rth         The Message Passing Paradigm  Message Passing Overview  The Cluster Model  Performance Issues  Rmpi  Example: Pipelined Method for Finding Primes  Memory Allocation Issues  Message-Passing Performance Subtleties         MapReduce Computation  Apache Hadoop  Other MapReduce Systems  R Interfaces to MapReduce Systems  An Alternative: "Snowdoop"         Parallel Sorting and Merging  The Elusive Goal of Optimality  Sorting Algorithms  Example: Bucket Sort in R  Example: Quicksort in OpenMP  Sorting in Rth  Some Timing Comparisons  Sorting on Distributed Data         Parallel Prefix Scan  General Formulation  Applications  General Strategies for Parallel Scan Computation  Implementations of Parallel Prefix Scan  Parallel cumsum() with OpenMP  Example: Moving Average         Parallel Matrix Operations  Tiled Matrices  Example: Snowdoop Approach to Matrix Operations  Parallel Matrix Multiplication BLAS Libraries  Example: A Look at the Performance of OpenBLAS  Example: Graph Connectedness  Solving Systems of Linear Equations  Sparse Matrices         Inherently Statistical Approaches: Subset Methods  Chunk Averaging  Bag of Little Bootstraps  Subsetting Variables        Appendix A: Review of Matrix Algebra  Appendix B: R Quick Start  Appendix C: Introduction to C for R Programmers