Stream Processing with Apache Spark: Mastering Structured Streaming and Spark Streaming

Copyright
Table of Contents
Foreword
Preface
	Who Should Read This Book?
	Installing Spark
	Learning Scala
	The Way Ahead
	Bibliography
	Conventions Used in This Book
	Using Code Examples
	O’Reilly Online Learning
	How to Contact Us
	Acknowledgments
		From Gerard
		From François
Part I. Fundamentals of Stream Processing with Apache Spark
	Chapter 1. Introducing Stream Processing
		What Is Stream Processing?
			Batch Versus Stream Processing
			The Notion of Time in Stream Processing
			The Factor of Uncertainty
		Some Examples of Stream Processing
		Scaling Up Data Processing
			MapReduce
			The Lesson Learned: Scalability and Fault Tolerance
		Distributed Stream Processing
			Stateful Stream Processing in a Distributed System
		Introducing Apache Spark
			The First Wave: Functional APIs
			The Second Wave: SQL
			A Unified Engine
			Spark Components
			Spark Streaming
			Structured Streaming
		Where Next?
	Chapter 2. Stream-Processing Model
		Sources and Sinks
		Immutable Streams Defined from One Another
		Transformations and Aggregations
		Window Aggregations
			Tumbling Windows
			Sliding Windows
		Stateless and Stateful Processing
		Stateful Streams
		An Example: Local Stateful Computation in Scala
			A Stateless Definition of the Fibonacci Sequence as a Stream Transformation
		Stateless or Stateful Streaming
		The Effect of Time
			Computing on Timestamped Events
			Timestamps as the Provider of the Notion of Time
			Event Time Versus Processing Time
			Computing with a Watermark
		Summary
	Chapter 3. Streaming Architectures
		Components of a Data Platform
		Architectural Models
		The Use of a Batch-Processing Component in a Streaming Application
		Referential Streaming Architectures
			The Lambda Architecture
			The Kappa Architecture
		Streaming Versus Batch Algorithms
			Streaming Algorithms Are Sometimes Completely Different in Nature
			Streaming Algorithms Can’t Be Guaranteed to Measure Well Against Batch Algorithms
		Summary
	Chapter 4. Apache Spark as a Stream-Processing Engine
		The Tale of Two APIs
		Spark’s Memory Usage
			Failure Recovery
			Lazy Evaluation
			Cache Hints
		Understanding Latency
		Throughput-Oriented Processing
		Spark’s Polyglot API
		Fast Implementation of Data Analysis
		To Learn More About Spark
		Summary
	Chapter 5. Spark’s Distributed Processing Model
		Running Apache Spark with a Cluster Manager
			Examples of Cluster Managers
		Spark’s Own Cluster Manager
		Understanding Resilience and Fault Tolerance in a Distributed System
			Fault Recovery
			Cluster Manager Support for Fault Tolerance
		Data Delivery Semantics
		Microbatching and One-Element-at-a-Time
			Microbatching: An Application of Bulk-Synchronous Processing
			One-Record-at-a-Time Processing
			Microbatching Versus One-at-a-Time: The Trade-Offs
		Bringing Microbatch and One-Record-at-a-Time Closer Together
		Dynamic Batch Interval
		Structured Streaming Processing Model
			The Disappearance of the Batch Interval
	Chapter 6. Spark’s Resilience Model
		Resilient Distributed Datasets in Spark
		Spark Components
		Spark’s Fault-Tolerance Guarantees
			Task Failure Recovery
			Stage Failure Recovery
			Driver Failure Recovery
		Summary
Appendix A. References for Part I
Part II. Structured Streaming
	Chapter 7. Introducing Structured Streaming
		First Steps with Structured Streaming
		Batch Analytics
		Streaming Analytics
			Connecting to a Stream
			Preparing the Data in the Stream
			Operations on Streaming Dataset
			Creating a Query
			Start the Stream Processing
			Exploring the Data
		Summary
	Chapter 8. The Structured Streaming Programming Model
		Initializing Spark
		Sources: Acquiring Streaming Data
			Available Sources
		Transforming Streaming Data
			Streaming API Restrictions on the DataFrame API
		Sinks: Output the Resulting Data
			format
			outputMode
			queryName
			option
			options
			trigger
			start()
		Summary
	Chapter 9. Structured Streaming in Action
		Consuming a Streaming Source
		Application Logic
		Writing to a Streaming Sink
		Summary
	Chapter 10. Structured Streaming Sources
		Understanding Sources
			Reliable Sources Must Be Replayable
			Sources Must Provide a Schema
		Available Sources
		The File Source
			Specifying a File Format
			Common Options
			Common Text Parsing Options (CSV, JSON)
			JSON File Source Format
			CSV File Source Format
			Parquet File Source Format
			Text File Source Format
		The Kafka Source
			Setting Up a Kafka Source
			Selecting a Topic Subscription Method
			Configuring Kafka Source Options
			Kafka Consumer Options
		The Socket Source
			Configuration
			Operations
		The Rate Source
			Options
	Chapter 11. Structured Streaming Sinks
		Understanding Sinks
		Available Sinks
			Reliable Sinks
			Sinks for Experimentation
			The Sink API
			Exploring Sinks in Detail
		The File Sink
			Using Triggers with the File Sink
			Common Configuration Options Across All Supported File Formats
			Common Time and Date Formatting (CSV, JSON)
			The CSV Format of the File Sink
			The JSON File Sink Format
			The Parquet File Sink Format
			The Text File Sink Format
		The Kafka Sink
			Understanding the Kafka Publish Model
			Using the Kafka Sink
		The Memory Sink
			Output Modes
		The Console Sink
			Options
			Output Modes
		The Foreach Sink
			The ForeachWriter Interface
			TCP Writer Sink: A Practical ForeachWriter Example
			The Moral of this Example
			Troubleshooting ForeachWriter Serialization Issues
	Chapter 12. Event Time–Based Stream Processing
		Understanding Event Time in Structured Streaming
		Using Event Time
		Processing Time
		Watermarks
		Time-Based Window Aggregations
			Defining Time-Based Windows
			Understanding How Intervals Are Computed
			Using Composite Aggregation Keys
			Tumbling and Sliding Windows
		Record Deduplication
		Summary
	Chapter 13. Advanced Stateful Operations
		Example: Car Fleet Management
		Understanding Group with State Operations
			Internal State Flow
		Using MapGroupsWithState
		Using FlatMapGroupsWithState
			Output Modes
			Managing State Over Time
		Summary
	Chapter 14. Monitoring Structured Streaming Applications
		The Spark Metrics Subsystem
			Structured Streaming Metrics
		The StreamingQuery Instance
			Getting Metrics with StreamingQueryProgress
		The StreamingQueryListener Interface
			Implementing a StreamingQueryListener
	Chapter 15. Experimental Areas: Continuous Processing and Machine Learning
		Continuous Processing
			Understanding Continuous Processing
			Using Continuous Processing
			Limitations
		Machine Learning
			Learning Versus Exploiting
			Applying a Machine Learning Model to a Stream
			Example: Estimating Room Occupancy by Using Ambient Sensors
			Online Training
Appendix B. References for Part II
Part III. Spark Streaming
	Chapter 16. Introducing Spark Streaming
		The DStream Abstraction
			DStreams as a Programming Model
			DStreams as an Execution Model
		The Structure of a Spark Streaming Application
			Creating the Spark Streaming Context
			Defining a DStream
			Defining Output Operations
			Starting the Spark Streaming Context
			Stopping the Streaming Process
		Summary
	Chapter 17. The Spark Streaming Programming Model
		RDDs as the Underlying Abstraction for DStreams
		Understanding DStream Transformations
		Element-Centric DStream Transformations
		RDD-Centric DStream Transformations
		Counting
		Structure-Changing Transformations
		Summary
	Chapter 18. The Spark Streaming Execution Model
		The Bulk-Synchronous Architecture
		The Receiver Model
			The Receiver API
			How Receivers Work
			The Receiver’s Data Flow
			The Internal Data Resilience
			Receiver Parallelism
			Balancing Resources: Receivers Versus Processing Cores
			Achieving Zero Data Loss with the Write-Ahead Log
		The Receiverless or Direct Model
		Summary
	Chapter 19. Spark Streaming Sources
		Types of Sources
			Basic Sources
			Receiver-Based Sources
			Direct Sources
		Commonly Used Sources
		The File Source
			How It Works
		The Queue Source
			How It Works
			Using a Queue Source for Unit Testing
			A Simpler Alternative to the Queue Source: The ConstantInputDStream
		The Socket Source
			How It Works
		The Kafka Source
			Using the Kafka Source
			How It Works
		Where to Find More Sources
	Chapter 20. Spark Streaming Sinks
		Output Operations
		Built-In Output Operations
			print
			saveAsxyz
			foreachRDD
		Using foreachRDD as a Programmable Sink
		Third-Party Output Operations
	Chapter 21. Time-Based Stream Processing
		Window Aggregations
		Tumbling Windows
			Window Length Versus Batch Interval
		Sliding Windows
			Sliding Windows Versus Batch Interval
			Sliding Windows Versus Tumbling Windows
		Using Windows Versus Longer Batch Intervals
		Window Reductions
			reduceByWindow
			reduceByKeyAndWindow
			countByWindow
			countByValueAndWindow
		Invertible Window Aggregations
		Slicing  Streams
		Summary
	Chapter 22. Arbitrary Stateful Streaming Computation
		Statefulness at the Scale of a Stream
		updateStateByKey
		Limitation of updateStateByKey
			Performance
			Memory Usage
		Introducing Stateful Computation with mapwithState
		Using mapWithState
		Event-Time Stream Computation Using mapWithState
	Chapter 23. Working with Spark SQL
		Spark SQL
		Accessing Spark SQL Functions from Spark Streaming
			Example: Writing Streaming Data to Parquet
		Dealing with Data at Rest
			Using Join to Enrich the Input Stream
		Join Optimizations
		Updating Reference Datasets in a Streaming Application
			Enhancing Our Example with a Reference Dataset
		Summary
	Chapter 24. Checkpointing
		Understanding the Use of Checkpoints
		Checkpointing DStreams
		Recovery from a Checkpoint
			Limitations
		The Cost of Checkpointing
		Checkpoint Tuning
	Chapter 25. Monitoring Spark Streaming
		The Streaming UI
		Understanding Job Performance Using the Streaming UI
			Input Rate Chart
			Scheduling Delay Chart
			Processing Time Chart
			Total Delay Chart
			Batch Details
		The Monitoring REST API
			Using the Monitoring REST API
			Information Exposed by the Monitoring REST API
		The Metrics Subsystem
		The Internal Event Bus
			Interacting with the Event Bus
		Summary
	Chapter 26. Performance Tuning
		The Performance Balance of Spark Streaming
			The Relationship Between Batch Interval and Processing Delay
			The Last Moments of a Failing Job
			Going Deeper: Scheduling Delay and Processing Delay
			Checkpoint Influence in Processing Time
		External Factors that Influence the Job’s Performance
		How to Improve Performance?
		Tweaking the Batch Interval
		Limiting the Data Ingress with Fixed-Rate Throttling
		Backpressure
		Dynamic Throttling
			Tuning the Backpressure PID
			Custom Rate Estimator
			A Note on Alternative Dynamic Handling Strategies
		Caching
		Speculative Execution
Appendix C. References for Part III
Part IV. Advanced Spark Streaming Techniques
	Chapter 27. Streaming Approximation and Sampling Algorithms
		Exactness, Real Time, and Big Data
			Exactness
			Real-Time Processing
			Big Data
		The Exactness, Real-Time, and Big Data triangle
			Big Data and Real Time
		Approximation Algorithms
		Hashing and Sketching: An Introduction
		Counting Distinct Elements: HyperLogLog
			Role-Playing Exercise: If We Were a System Administrator
			Practical HyperLogLog in Spark
		Counting Element Frequency: Count Min Sketches
			Introducing Bloom Filters
			Bloom Filters with Spark
			Computing Frequencies with a Count-Min Sketch
		Ranks and Quantiles: T-Digest
			T-Digest in Spark
		Reducing the Number of Elements: Sampling
			Random Sampling
			Stratified Sampling
	Chapter 28. Real-Time Machine Learning
		Streaming Classification with Naive Bayes
			streamDM Introduction
			Naive Bayes in Practice
			Training a Movie Review Classifier
		Introducing Decision Trees
		Hoeffding Trees
			Hoeffding Trees in Spark, in Practice
		Streaming Clustering with Online K-Means
			K-Means Clustering
			Online Data and K-Means
			The Problem of Decaying Clusters
			Streaming K-Means with Spark Streaming
Appendix D. References for Part IV
Part V. Beyond Apache Spark
	Chapter 29. Other Distributed Real-Time Stream Processing Systems
		Apache Storm
			Processing Model
			The Storm Topology
			The Storm Cluster
			Compared to Spark
		Apache Flink
			A Streaming-First Framework
			Compared to Spark
		Kafka Streams
			Kafka Streams Programming Model
			Compared to Spark
		In the Cloud
			Amazon Kinesis on AWS
			Microsoft Azure Stream Analytics
			Apache Beam/Google Cloud Dataflow
	Chapter 30. Looking Ahead
		Stay Plugged In
			Seek Help on Stack Overflow
			Start Discussions on the Mailing Lists
			Attend Conferences
		Attend Meetups
			Read Books
		Contributing to the Apache Spark Project
Appendix E. References for Part V
Index
About the Authors
Colophon