Mastering Kafka Streams and ksqlDB

Cover
Copyright
Table of Contents
Foreword
Preface
	Who Should Read This Book
	Navigating This Book
	Source Code
	Kafka Streams Version
	ksqlDB Version
	Conventions Used in This Book
	Using Code Examples
	O’Reilly Online Learning
	How to Contact Us
	Acknowledgments
Part I. Kafka
	Chapter 1. A Rapid Introduction to Kafka
		Communication Model
		How Are Streams Stored?
		Topics and Partitions
		Events
		Kafka Cluster and Brokers
		Consumer Groups
		Installing Kafka
		Hello, Kafka
		Summary
Part II. Kafka Streams
	Chapter 2. Getting Started with Kafka Streams
		The Kafka Ecosystem
			Before Kafka Streams
			Enter Kafka Streams
		Features at a Glance
		Operational Characteristics
			Scalability
			Reliability
			Maintainability
		Comparison to Other Systems
			Deployment Model
			Processing Model
			Kappa Architecture
		Use Cases
		Processor Topologies
			Sub-Topologies
			Depth-First Processing
			Benefits of Dataflow Programming
			Tasks and Stream Threads
		High-Level DSL Versus Low-Level Processor API
		Introducing Our Tutorial: Hello, Streams
			Project Setup
			Creating a New Project
			Adding the Kafka Streams Dependency
			DSL
			Processor API
		Streams and Tables
			Stream/Table Duality
			KStream,  KTable, GlobalKTable
		Summary
	Chapter 3. Stateless Processing
		Stateless Versus Stateful Processing
		Introducing Our Tutorial: Processing a Twitter Stream
		Project Setup
		Adding a KStream Source Processor
		Serialization/Deserialization
			Building a Custom Serdes
			Defining Data Classes
			Implementing a Custom Deserializer
			Implementing a Custom Serializer
			Building the Tweet Serdes
		Filtering Data
		Branching Data
		Translating Tweets
		Merging Streams
		Enriching Tweets
			Avro Data Class
			Sentiment Analysis
		Serializing Avro Data
			Registryless Avro Serdes
			Schema Registry–Aware Avro Serdes
		Adding a Sink Processor
		Running the Code
		Empirical Verification
		Summary
	Chapter 4. Stateful Processing
		Benefits of Stateful Processing
		Preview of Stateful Operators
		State Stores
			Common Characteristics
			Persistent Versus In-Memory Stores
		Introducing Our Tutorial: Video Game Leaderboard
		Project Setup
		Data Models
		Adding the Source Processors
			KStream
			KTable
			GlobalKTable
		Registering Streams and Tables
		Joins
			Join Operators
			Join Types
			Co-Partitioning
			Value Joiners
			KStream to KTable Join (players Join)
			KStream to GlobalKTable Join (products Join)
		Grouping Records
			Grouping Streams
			Grouping Tables
		Aggregations
			Aggregating Streams
			Aggregating Tables
		Putting It All Together
		Interactive Queries
			Materialized Stores
			Accessing Read-Only State Stores
			Querying Nonwindowed Key-Value Stores
			Local Queries
			Remote Queries
		Summary
	Chapter 5. Windows and Time
		Introducing Our Tutorial: Patient Monitoring Application
		Project Setup
		Data Models
		Time Semantics
		Timestamp Extractors
			Included Timestamp Extractors
			Custom Timestamp Extractors
			Registering Streams with a Timestamp Extractor
		Windowing Streams
			Window Types
			Selecting a Window
			Windowed Aggregation
		Emitting Window Results
			Grace Period
			Suppression
		Filtering and Rekeying Windowed KTables
		Windowed Joins
		Time-Driven Dataflow
			Alerts Sink
			Querying Windowed Key-Value Stores
		Summary
	Chapter 6. Advanced State Management
		Persistent Store Disk Layout
		Fault Tolerance
			Changelog Topics
			Standby Replicas
		Rebalancing: Enemy of the State (Store)
		Preventing State Migration
			Sticky Assignment
			Static Membership
		Reducing the Impact of Rebalances
			Incremental Cooperative Rebalancing
			Controlling State Size
		Deduplicating Writes with Record Caches
		State Store Monitoring
			Adding State Listeners
			Adding State Restore Listeners
		Built-in Metrics
		Interactive Queries
		Custom State Stores
		Summary
	Chapter 7. Processor API
		When to Use the Processor API
		Introducing Our Tutorial: IoT Digital Twin Service
		Project Setup
		Data Models
		Adding Source Processors
		Adding Stateless Stream Processors
		Creating Stateless Processors
		Creating Stateful Processors
		Periodic Functions with Punctuate
		Accessing Record Metadata
		Adding Sink Processors
		Interactive Queries
		Putting It All Together
		Combining the Processor API with the DSL
		Processors and Transformers
		Putting It All Together: Refactor
		Summary
Part III. ksqlDB
	Chapter 8. Getting Started with ksqlDB
		What Is ksqlDB?
		When to Use ksqlDB
		Evolution of a New Kind of Database
			Kafka Streams Integration
			Connect Integration
			How Does ksqlDB Compare to a Traditional SQL Database?
			Similarities
			Differences
		Architecture
			ksqlDB Server
			ksqlDB Clients
		Deployment Modes
			Interactive Mode
			Headless Mode
		Tutorial
			Installing ksqlDB
			Running a ksqlDB Server
			Precreating Topics
			Using the ksqlDB CLI
			Summary
	Chapter 9. Data Integration with ksqlDB
		Kafka Connect Overview
		External Versus Embedded Connect
			External Mode
			Embedded Mode
		Configuring Connect Workers
			Converters and Serialization Formats
		Tutorial
		Installing Connectors
			Creating Connectors with ksqlDB
			Showing Connectors
			Describing Connectors
			Dropping Connectors
		Verifying the Source Connector
		Interacting with the Kafka Connect Cluster Directly
		Introspecting Managed Schemas
		Summary
	Chapter 10. Stream Processing Basics with ksqlDB
		Tutorial: Monitoring Changes at Netflix
		Project Setup
		Source Topics
		Data Types
			Custom Types
		Collections
			Creating Source Collections
			With Clause
		Working with Streams and Tables
			Showing Streams and Tables
			Describing Streams and Tables
			Altering Streams and Tables
			Dropping Streams and Tables
		Basic Queries
			Insert Values
			Simple Selects (Transient Push Queries)
			Projection
			Filtering
			Flattening/Unnesting Complex Structures
		Conditional Expressions
			Coalesce
			IFNULL
			Case Statements
		Writing Results Back to Kafka (Persistent Queries)
			Creating Derived Collections
		Putting It All Together
		Summary
	Chapter 11. Intermediate and Advanced Stream Processing with ksqlDB
		Project Setup
		Bootstrapping an Environment from a SQL File
		Data Enrichment
			Joins
			Windowed Joins
		Aggregations
			Aggregation Basics
			Windowed Aggregations
		Materialized Views
		Clients
		Pull Queries
			Curl
		Push Queries
			Push Queries via Curl
		Functions and Operators
			Operators
			Showing Functions
			Describing Functions
			Creating Custom Functions
			Additional Resources for Custom ksqlDB Functions
		Summary
Part IV. The Road to Production
	Chapter 12. Testing, Monitoring, and Deployment
		Testing
			Testing ksqlDB Queries
			Testing Kafka Streams
			Behavioral Tests
			Benchmarking
			Kafka Cluster Benchmarking
			Final Thoughts on Testing
		Monitoring
			Monitoring Checklist
			Extracting JMX Metrics
		Deployment
			ksqlDB Containers
			Kafka Streams Containers
			Container Orchestration
		Operations
			Resetting a Kafka Streams Application
			Rate-Limiting the Output of Your Application
			Upgrading Kafka Streams
		Upgrading ksqlDB
		Summary
Appendix A. Kafka Streams Configuration
	Configuration Management
	Configuration Properties
		Consumer-Specific Configurations
Appendix B. ksqlDB Configuration
	Query Configurations
	Server Configurations
	Security Configurations
Index
About the Author
Colophon