Learning Spark: Lightning-Fast Data Analytics

Copyright......Page 6
Table of Contents......Page 7
Foreword......Page 15
Who This Book Is For......Page 17
How the Book Is Organized......Page 18
Conventions Used in This Book......Page 20
O’Reilly Online Learning......Page 21
Acknowledgments......Page 22
Big Data and Distributed Computing at Google......Page 25
Hadoop at Yahoo!......Page 26
Spark’s Early Years at AMPLab......Page 27
Speed......Page 28
Extensibility......Page 29
Apache Spark Components as a Unified Stack......Page 30
Apache Spark’s Distributed Execution......Page 34
Who Uses Spark, and for What?......Page 38
Community Adoption and Expansion......Page 40
Step 1: Downloading Apache Spark......Page 43
Spark’s Directories and Files......Page 45
Step 2: Using the Scala or PySpark Shell......Page 46
Using the Local Machine......Page 47
Step 3: Understanding Spark Application Concepts......Page 49
Spark Application and SparkSession......Page 50
Spark Jobs......Page 51
Transformations, Actions, and Lazy Evaluation......Page 52
Narrow and Wide Transformations......Page 54
The Spark UI......Page 55
Your First Standalone Application......Page 58
Counting M&Ms for the Cookie Monster......Page 59
Building Standalone Applications in Scala......Page 64
Summary......Page 66
Spark: What’s Underneath an RDD?......Page 67
Structuring Spark......Page 68
Key Merits and Benefits......Page 69
The DataFrame API......Page 71
Spark’s Basic Data Types......Page 72
Spark’s Structured and Complex Data Types......Page 73
Schemas and Creating DataFrames......Page 74
Columns and Expressions......Page 78
Rows......Page 81
Common DataFrame Operations......Page 82
End-to-End DataFrame Example......Page 92
Typed Objects, Untyped Objects, and Generic Rows......Page 93
Creating Datasets......Page 95
Dataset Operations......Page 96
DataFrames Versus Datasets......Page 98
When to Use RDDs......Page 99
Spark SQL and the Underlying Engine......Page 100
The Catalyst Optimizer......Page 101
Summary......Page 106
Chapter 4. Spark SQL and DataFrames: Introduction to Built-in Data Sources......Page 107
Using Spark SQL in Spark Applications......Page 108
Basic Query Examples......Page 109
Managed Versus UnmanagedTables......Page 113
Creating SQL Databases and Tables......Page 114
Creating Views......Page 115
Reading Tables into DataFrames......Page 117
DataFrameReader......Page 118
DataFrameWriter......Page 120
Parquet......Page 121
JSON......Page 124
CSV......Page 126
Avro......Page 128
ORC......Page 130
Images......Page 132
Binary Files......Page 134
Summary......Page 135
Spark SQL and Apache Hive......Page 137
User-Defined Functions......Page 138
Using the Spark SQL Shell......Page 143
Working with Beeline......Page 144
Working with Tableau......Page 146
JDBC and SQL Databases......Page 153
PostgreSQL......Page 156
MySQL......Page 157
Azure Cosmos DB......Page 158
MS SQL Server......Page 160
Other External Sources......Page 161
Option 2: User-Defined Function......Page 162
Built-in Functions for Complex Data Types......Page 163
Higher-Order Functions......Page 165
Common DataFrames and Spark SQL Operations......Page 168
Unions......Page 171
Joins......Page 172
Windowing......Page 173
Modifications......Page 175
Summary......Page 179
Single API for Java and Scala......Page 181
Scala Case Classes and JavaBeans for Datasets......Page 182
Creating Sample Data......Page 184
Transforming Sample Data......Page 186
Memory Management for Datasets and DataFrames......Page 191
Spark’s Internal Format Versus Java Object Format......Page 192
Serialization and Deserialization (SerDe)......Page 193
Strategies to Mitigate Costs......Page 194
Summary......Page 196
Viewing and Setting Apache Spark Configurations......Page 197
Scaling Spark for Large Workloads......Page 201
DataFrame.cache()......Page 207
DataFrame.persist()......Page 208
A Family of Spark Joins......Page 211
Broadcast Hash Join......Page 212
Shuffle Sort Merge Join......Page 213
Journey Through the Spark UI Tabs......Page 221
Summary......Page 229
Evolution of the Apache Spark Stream Processing Engine......Page 231
The Advent of Micro-Batch Stream Processing......Page 232
Lessons Learned from Spark Streaming (DStreams)......Page 233
The Philosophy of Structured Streaming......Page 234
The Programming Model of Structured Streaming......Page 235
Five Steps to Define a Streaming Query......Page 237
Under the Hood of an Active Streaming Query......Page 243
Recovering from Failures with Exactly-Once Guarantees......Page 245
Monitoring an Active Query......Page 247
Files......Page 250
Apache Kafka......Page 252
Custom Streaming Sources and Sinks......Page 254
Incremental Execution and Streaming State......Page 258
Stateful Transformations......Page 259
Aggregations Not Based on Time......Page 262
Aggregations with Event-Time Windows......Page 263
Stream–Static Joins......Page 270
Stream–Stream Joins......Page 272
Arbitrary Stateful Computations......Page 277
Modeling Arbitrary Stateful Operations with mapGroupsWithState()......Page 278
Using Timeouts to Manage Inactive Groups......Page 281
Generalization with flatMapGroupsWithState()......Page 285
Performance Tuning......Page 286
Summary......Page 288
The Importance of an Optimal Storage Solution......Page 289
A Brief Introduction to Databases......Page 290
Limitations of Databases......Page 291
A Brief Introduction to Data Lakes......Page 292
Reading from and Writing to Data Lakes using Apache Spark......Page 293
Limitations of Data Lakes......Page 294
Lakehouses: The Next Step in the Evolution of Storage Solutions......Page 295
Apache Iceberg......Page 296
Delta Lake......Page 297
Configuring Apache Spark with Delta Lake......Page 298
Loading Data into a Delta Lake Table......Page 299
Loading Data Streams into a Delta Lake Table......Page 301
Enforcing Schema on Write to Prevent Data Corruption......Page 302
Transforming Existing Data......Page 303
Auditing Data Changes with Operation History......Page 306
Querying Previous Snapshots of a Table with Time Travel......Page 307
Summary......Page 308
Chapter 10. Machine Learning with MLlib......Page 309
Supervised Learning......Page 310
Unsupervised Learning......Page 312
Designing Machine Learning Pipelines......Page 313
Data Ingestion and Exploration......Page 314
Creating Training and Test Data Sets......Page 315
Preparing Features with Transformers......Page 317
Understanding Linear Regression......Page 318
Using Estimators to Build Models......Page 319
Creating a Pipeline......Page 320
Evaluating Models......Page 326
Saving and Loading Models......Page 330
Tree-Based Models......Page 331
k-Fold Cross-Validation......Page 340
Optimizing Pipelines......Page 344
Summary......Page 345
Model Management......Page 347
MLflow......Page 348
Model Deployment Options with MLlib......Page 354
Batch......Page 356
Streaming......Page 357
Model Export Patterns for Real-Time Inference......Page 358
Pandas UDFs......Page 360
Spark for Distributed Hyperparameter Tuning......Page 361
Summary......Page 365
Dynamic Partition Pruning......Page 367
Adaptive Query Execution......Page 369
SQL Join Hints......Page 372
Catalog Plugin API and DataSourceV2......Page 373
Accelerator-Aware Scheduler......Page 375
Structured Streaming......Page 376
Redesigned Pandas UDFs with Python Type Hints......Page 378
Iterator Support in Pandas UDFs......Page 379
New Pandas Function APIs......Page 380
Changes to the DataFrame and Dataset APIs......Page 381
DataFrame and SQL Explain Commands......Page 382
Summary......Page 384
Index......Page 385
Colophon......Page 398