Hands-on Guide to Apache Spark 3: Build Scalable Computing Engines for Batch and Stream Data Processing

Table of Contents
About the Author
About the Technical Reviewer
Part I: Apache Spark Batch Data Processing
	Chapter 1: Introduction to Apache Spark for Large-Scale Data Analytics
		1.1 What Is Apache Spark?
			Simpler to Use and Operate
			Fast
			Scalable
			Ease of Use
			Fault Tolerance at Scale
		1.2 Spark Unified Analytics Engine
		1.3 How Apache Spark Works
			Spark Application Model
			Spark Execution Model
			Spark Cluster Model
		1.4 Apache Spark Ecosystem
			Spark Core
			Spark APIs
			Spark SQL and DataFrames and Datasets
			Spark Streaming
			Spark GraphX
		1.5 Batch vs. Streaming Data
			What Is Batch Data Processing?
			What Is Stream Data Processing?
			Difference Between Stream Processing and Batch Processing
		1.6 Summary
	Chapter 2: Getting Started with Apache Spark
		2.1 Downloading and Installing Apache Spark
			Installation of Apache Spark on Linux
				Step 1: Verifying the Java Installation
				Step 2: Installing Spark
				Step 3: Moving Spark Software Files
				Step 4: Setting Up the Environment for Spark
				Step 5: Verifying the Spark Installation
			Installation of Apache Spark on Windows
				Step 1: Java Installation
				Step 2: Download Apache Spark
				Step 3: Install Apache Spark
				Step 4: Download the winutils File for Hadoop
				Step 5: Configure System and Environment Variables
		2.2 Hands-On Spark Shell
			Using the Spark Shell Command
				The Scala Shell Command Line
				The Pyspark Shell Command Line
			Running Self-Contained Applications with the spark-submit Command
				Spark Submit Options
				Deployment Mode Options
				Cluster Manager Options
				Tuning Resource Allocation
				Dynamically Loading Spark Submit Configurations
				Application Properties
				Runtime Environment
				Dynamic Allocation
				Others
		2.3 Spark Application Concepts
			Spark Application and SparkSession
			Access the Existing SparkSession
				SparkSession in spark-shell
				Create a SparkSession Programmatically
		2.4 Transformations, Actions, Immutability, and Lazy Evaluation
			Transformations
			Narrow Transformations
			Wide Transformations
			Actions
		2.5 Summary
	Chapter 3: Spark Low-Level API
		3.1	 Resilient Distributed Datasets (RDDs)
			Creating RDDs from Parallelized Collections
			Creating RDDs from External Datasets
			Creating RDDs from Existing RDDs
		3.2	 Working with Key-Value Pairs
			Creating Pair RDDs
			Showing the Distinct Keys of a Pair RDD
			Transformations on Pair RDDs
				Sorting Pair RDDs by Key
				Adding Values by Key in a RDD
				Saving a RDD as a Text File
				Combining Data at Each Partition
				Merging Values with a Neutral ZeroValue
				Combining Elements by Key Using Custom Aggregation Functions
				Grouping of Data on Pair RDDs
					Performance Considerations of groupByKey
				Joins on Pair RDDs
					Returning the Keys Present in Both RDDs
					Returning the Keys Present in the Source RDD
					Returning the Keys Present in the Parameter RDD
				Sorting Data on Pair RDDs
				Sorting a RDD by a Given Key Function
			Actions on Pair RDDs
				Count RDD Instances by Key
				Count RDD Instances by Value
				Returning Key-Value Pairs as a Dictionary
				Collecting All Values Associated With a Key
		3.3	 Spark Shared Variables: Broadcasts and Accumulators
			Broadcast Variables
				When to Use Broadcast Variables
				How to Create a Broadcast Variable
			Accumulators
		3.4	 When to Use RDDs
		3.5	 Summary
	Chapter 4: The Spark High-Level APIs
		4.1	 Spark Dataframes
			Attributes of Spark DataFrames
			Methods for Creating Spark DataFrames
				Manually Creating a Spark DataFrame Using toDF()
				Manually Creating a Spark DataFrame Using createDataFrame()
				Data Sources for Creating Spark DataFrames
				Querying Files Using SQL
				Ignoring Corrupt and Missing Files
				Time-Based Paths
				Specifying Save Options
				NOTICE
				Read and Write Apache Parquet Files
				Saving and Data Compression of a DataFrame to a Parquet File Format
				Direct Queries on Parquet Files
				Parquet File Partitioning
				Reading Parquet File Partitions
				Read and Write JSON Files with Spark
				Reading Multiple JSON Files at Once
				Reading JSON Files Based on Patterns at Once
				Direct Queries on JSON Files
				Saving a DataFrame to a JSON File
				Saving Modes
				Load JSON Files Based on Customized Schemas
				Work with Complex Nested JSON Structures Using Spark
				Read and Write CSV Files with Spark
				Read and Write Hive Tables
				Read and Write Data via JDBC from and to Databases
		4.2	 Use of Spark DataFrames
			Select DataFrame Columns
				Selecting All or Specific DataFrame Columns
			Select Columns Based on Name Patterns
			Filtering Results of a Query Based on One or Multiple Conditions
			Using Different Column Name Notations
			Using Logical Operators for Multi-condition Filtering
			Manipulating Spark DataFrame Columns
			Renaming DataFrame Columns
			Dropping DataFrame Columns
			Creating a New Dataframe Column Dependent on Another Column
			User-Defined Functions (UDFs)
			Merging DataFrames with Union and UnionByName
				Merging DataFrames with Duplicate Values
			Joining DataFrames with Join
		4.3	 Spark Cache and Persist of Data
			Unpersisting Cached Data
		4.4	 Summary
	Chapter 5: Spark Dataset API and Adaptive Query Execution
		5.1	 What Are Spark Datasets?
		5.2	 Methods for Creating Spark Datasets
		5.3	 Adaptive Query Execution
		5.4	 Data-Dependent Adaptive Determination of the Shuffle Partition Number
		5.5	 Runtime Replanning of Join Strategies
		5.6	 Optimization of Unevenly Distributed Data Joins
		5.7	 Enabling the Adaptive Query Execution (AQE)
		5.8	 Summary
	Chapter 6: Introduction to Apache Spark Streaming
		6.1 Real-Time Analytics of Bound and Unbound Data
		6.2 Challenges of Stream Processing
		6.3 The Uncertainty Component of Data Streams
		6.4 Apache Spark Streaming’s Execution Model
		6.5 Stream Processing Architectures
			The Lambda Architecture
				Data Ingestion Layer
				Batch Processing Layer
				Real-Time or Speed Layer
				Serving Layer
				Pros and Cons of the Lambda Architecture
			The Kappa Architecture
				Pros and Cons of the Kappa Architecture
		6.6 Spark Streaming Architecture: Discretized Streams
		6.7 Spark Streaming Sources and Receivers
			Basic Input Sources
				Socket Connection Streaming Sources
				Running Socket Streaming Applications Locally
				Improving Our Data Analytics with Spark Streaming Transformations
				File System Streaming Sources
				How Spark Monitors File Systems
				Running File System Streaming Applications Locally
				Known Issues While Dealing with Object Stores Data Sources
			Advanced Input Sources
		6.8 Spark Streaming Graceful Shutdown
		6.9 Transformations on DStreams
		6.10 Summary
Part II: Apache Spark Streaming
	Chapter 7: Spark Structured Streaming
		7.1	 General Rules for Message Delivery Reliability
		7.2	 Structured Streaming vs. Spark Streaming
		7.3	 What Is Apache Spark Structured Streaming?
			Spark Structured Streaming Input Table
			Spark Structured Streaming Result Table
			Spark Structured Streaming Output Modes
		7.4	 Datasets and DataFrames Streaming API
			Socket Structured Streaming Sources
			Running Socket Structured Streaming Applications Locally
			File System Structured Streaming Sources
			Running File System Streaming Applications Locally
		7.5	 Spark Structured Streaming Transformations
			Streaming State in Spark Structured Streaming
			Spark Stateless Streaming
			Spark Stateful Streaming
			Stateful Streaming Aggregations
				Time-Based Aggregations
				No-Time-Based Aggregations
		7.6	 Spark Checkpointing Streaming
			Recovering from Failures with Checkpointing
		7.7	 Summary
	Chapter 8: Streaming Sources and Sinks
		8.1	 Spark Streaming Data Sources
			Reading Streaming Data from File Data Sources
			Reading Streaming Data from Kafka
			Reading Streaming Data from MongoDB
		8.2	 Spark Streaming Data Sinks
			Writing Streaming Data to the Console Sink
			Writing Streaming Data to the File Sink
			Writing Streaming Data to the Kafka Sink
			Writing Streaming Data to the ForeachBatch Sink
			Writing Streaming Data to the Foreach Sink
			Writing Streaming Data to Other Data Sinks
		8.3	 Summary
	Chapter 9: Event-Time Window Operations and Watermarking
		9.1	 Event-Time Processing
		9.2	 Stream Temporal Windows in Apache Spark
			What Are Temporal Windows and Why Are They Important in Streaming
		9.3	 Tumbling Windows
		9.4	 Sliding Windows
		9.5	 Session Windows
			Session Window with Dynamic Gap
		9.6	 Watermarking in Spark Structured Streaming
			What Is a Watermark?
		9.7	 Summary
	Chapter 10: Future Directions for Spark Streaming
		10.1 Streaming Machine Learning with Spark
			What Is Logistic Regression?
			Types of Logistic Regression
			Use Cases of Logistic Regression
			Assessing the Sensitivity and Specificity of Our Streaming ML Model
		10.2 Spark 3.3.x
			Spark RocksDB State Store Database
				What Is RocksDB?
					BlockBasedTableConfig
					RocksDB Possible Performance Degradation
		10.3 The Project Lightspeed
			Predictable Low Latency
			Enhanced Functionality for Processing Data/Events
			New Ecosystem of Connectors
			Improve Operations and Troubleshooting
		10.4 Summary
Bibliography
Index