Streaming Databases: Unifying Batch and Stream Processing

Cover
Copyright
Table of Contents
Foreword
Preface
	Conventions Used in This Book
	Using Code Examples
	O’Reilly Online Learning
	How to Contact Us
	Hubert’s Acknowledgements
	Ralph’s Acknowledgments
Chapter 1. Streaming Foundations
	Turning the Database Inside Out
	Externalizing Database Features
		Write-Ahead Log
		Streaming Platforms
		Materialized Views
	Use Case: Clickstream Analysis
		Understanding Transactions and Events
		Domain-Driven Design
	Context Enrichment
	Change Data Capture
	Connectors
		Connector Middleware
		Embedded
		Custom-Built
	Summary
Chapter 2. Stream Processing Platforms
	Stateful Transformations
	Data Pipelines
		ELT Limitations
		Stream Processing with ELT
	Stream Processors
		Popular Stream Processors
		Newer Stream Processors
	Emulating Materialized Views in Apache Spark
	Two Types of Streams
		Append Stream
		Debezium Change Data
		Materialized Views
	Summary
Chapter 3. Serving Real-Time Data
	Real-Time Expectations
	Choosing an Analytical Data Store
	Sourcing from a Topic
	Ingestion Transformations
	OLTP Versus OLAP
		ACID
		Row- Versus Column-Based Optimization
	Queries Per Second and Concurrency
	Indexing
	Serving Analytical Results
		Synchronous Queries
		Asynchronous Queries
		Push Versus Pull Queries
	Summary
Chapter 4. Materialized Views
	Views, Materialized Views, and Incremental Updates
	Change Data Capture
	Push Versus Pull Queries
	CDC and Upsert
	Joining Streams
		Apache Calcite
		Clickstream Use Case
	Summary
Chapter 5. Introduction to Streaming Databases
	Identifying the Streaming Database
		Column-Based Streaming Database
		Row-Based Streaming Database
		Edge Streaming-Like Databases
	SQL Expressivity
	Streaming Debuggability
		Advantages of Debugging in Streaming Databases
		SQL Is Not a Silver Bullet
	Streaming Database Implementations
	Streaming Database Architecture
	ELT with Streaming Databases
	Summary
Chapter 6. Consistency
	A Toy Example
		Transactions
		Analyzing the Transactions
	Comparing Consistency Across Stream Processing Systems
		Flink SQL
		ksqlDB
		Proton (Timeplus)
		RisingWave
		Materialize
		Pathway
		Out-of-Order Messages
	Going Beyond Eventual Consistency
		Why Do Eventually Consistent Stream Processors Fail the Toy Example?
		How Do Internally Consistent Stream Processing Systems Pass the Toy Example?
		How Can We Fix Eventually Consistent Stream Processing Systems to Pass the Toy Example?
	Consistency Versus Latency
	Summary
Chapter 7. Emergence of Other Hybrid Data Systems
	Data Planes
	Hybrid Transactional/Analytical Database
	Other Hybrid Databases
	Motivations for Hybrid Systems
	The Influence of PostgreSQL on Hybrid Databases
	Near-Edge Analytics
	Next-Generation Hybrid Databases
		Next-Generation Streaming OLTP Databases
		Next-Generation Streaming RTOLAP Databases
		Next-Generation HTAP Databases
	Summary
Chapter 8. Zero-ETL or Near-Zero-ETL
	ETL Model
	Zero-ETL
	Near-Zero-ETL
		PeerDB
		Proton
		Embedded OLAP
		Data Gravity and Replication
		Analytical Data Reduction
	Lambda Architecture
		Apache Pinot Hybrid Tables
		Pipeline Configurations
	Summary
Chapter 9. The Streaming Plane
	Data Gravity
	Components of the Streaming Plane
	Streaming Plane Infrastructure
	Operational Analytics
	Data Mesh
		Pillars of a Data Mesh
		Challenge of a Data Mesh
	Streaming Data Mesh with Streaming Plane and Streaming Databases
		Data Locality
		Data Replication
	Summary
Chapter 10. Deployment Models
	Consistent Streaming Database
	Consistent Streaming Processor and RTOLAP
	Eventually Consistent OLAP Streaming Database
	Eventually Consistent Stream Processor and RTOLAP
	Eventually Consistent Stream Processor and HTAP
	ksqlDB
	Incremental View Maintenance
	Postgres Multicorn Foreign Data Wrapper
	When to Use Code-Based Stream Processors
	When to Use Lakehouse/Streamhouse Technologies
	Caching Technologies
	Where to Do Processing and Querying in General?
		The Four “Where” Questions
		An Analytical Use Case
		Consequences
	Summary
Chapter 11. Future State of Real-Time Data
	The Convergence of the Data Planes
	Graph Databases
		Memgraph
		thatDot/Quine
	Vector Databases
		Milvus 2.x: Streaming as the Central Backbone
		RTOLAP Databases: Adding Vector Search
	Incremental View Maintenance
		pg_ivm
		Hydra
		Epsio
		Feldera
		PeerDB
	Data Wrapping and Postgres Multicorn
	Classical Databases
	Data Warehouses
		BigQuery
		Redshift
		Snowflake
	Lakehouse
		Delta Lake
		Apache Paimon
		Apache Iceberg
		Apache Hudi
		OneTable or XTable
		The Relationship of Streaming and Lakehouses
	Conclusion
Index
About the Authors
Colophon