Mastering Apache Pulsar: Cloud Native Event Streaming at Scale

Cover
Copyright
Table of Contents
Preface
	Why I Wrote This Book
	Who This Book Is For
	How I Organized This Book
	Conventions Used in This Book
	Using Code Examples
	O’Reilly Online Learning
	How to Contact Us
	Acknowledgments
Chapter 1. The Value of Real-Time Messaging
	Data in Motion
	Resource Efficiency
	Interesting Applications
		Banking
		Medical
		Security
		Internet of Things
	Summary
Chapter 2. Event Streams and Event Brokers
	Publish/Subscribe
	Queues
	Failure Modes
	Push Versus Poll
	The Need for Pulsar
		Unification
		Modularity
		Performance
	Summary
Chapter 3. Pulsar
	Origins of Pulsar
	Pulsar Design Principles
		Multitenancy
		Geo-Replication
		Performance
		Modularity
	Pulsar Ecosystem
		Pulsar Functions
		Pulsar IO
		Pulsar SQL
	Pulsar Success Stories
		Yahoo! JAPAN
		Splunk
		Iterable
	Summary
Chapter 4. Pulsar Internals
	Brokers
		Message Cache
		BookKeeper and ZooKeeper Communication
		Schema Validation
		Inter-Broker Communication
		Pulsar Functions and Pulsar IO
	Apache BookKeeper
		Write-Ahead Logging
		Message Storing
		Object/Blob Storage
		Pravega
		Majordodo
	Apache ZooKeeper
		Naming Service
		Configuration Management
		Leader Election
		Notification System
		Apache Kafka
		Apache Druid
	Pulsar Proxy
	Java Virtual Machine (JVM)
		Netty
		Apache Spark
		Apache Lucene
	Summary
Chapter 5. Consumers
	What Does It Mean to Be a Consumer?
	Subscriptions
		Exclusive
		Shared
		Key_Shared
		Failover
	Acknowledgments
		Individual Ack
		Cumulative Ack
	Schemas
		Consumer Schema Management
	Consumption Modes
		Batching
		Chunking
	Advanced Configuration
		Delayed Messages
		Retention Policy
		Backlog Quota
	Configuring a Consumer
		Replay
		Dead Letter Topics
		Retry Letter Topics
	Summary
Chapter 6. Producers
	Synchronous Producers
	Asynchronous Producers
	Producer Routing
		Round-Robin Routing
		Single Partition Routing
		Custom Partition Routing 
	Producer Configuration
		topicName
		producerName
		sendTimeoutMs
		blockIfQueueFull
		maxPendingMessages
		maxPendingMessagesAcrossPartitions
		messageRoutingMode
		hashingScheme
		cryptoFailureAction
		batchingMaxPublishDelayMicros
		batchingMaxMessages
		batchingEnabled
		compressionType
	Schema on Write
		Using the Schema Registry
	Nonpersistent Topics
		Use Cases
		Using Nonpersistent Topics
	Transactions
	Summary
Chapter 7. Pulsar IO
	Pulsar IO Architecture
		Runtime
		Performance Considerations
	Use Cases
		Simple Event Processing Pipelines
		Change Data Capture
	Considerations
		Message Serialization
		Pipeline Stability
		Failure Handling
	Examples
		Elasticsearch
		Netty
	Writing Your Connector
		TimescaleDB
	Summary
Chapter 8. Pulsar Functions
	Stream Processing
	Pulsar Functions Architecture
		Runtime
		Isolation
	Isolation with Kubernetes Function Deployments
	Use Cases
		Creating Pulsar Functions
		Simple Event Processing
		Topic Hygiene
		Topic Accounting
	Summary
Chapter 9. Tiered Storage
	Storing Data in the Cloud
		Object Storage
	Use Cases
		Replication
		CQRS
		Disaster Recovery
	Offloading Data
		Pulsar Offloaders
	Retrieving Offloaded Data
		Interacting with Object Store Data
		Repopulating Topics
		Utilizing Pulsar Client
	Summary
Chapter 10. Pulsar SQL
	Streams as Tables
	SQL-on-Anything Engines
		Apache Flink: An Alternative Perspective
		Presto/Trino
	How Pulsar SQL Works
	Configuring Pulsar SQL
	Performance Considerations
	Summary
Chapter 11. Deploying Pulsar
	Docker
	Bare Metal
		Minimum Requirements
		Getting Started
		Deploying ZooKeeper
		Starting BookKeeper
		Starting Pulsar
	Public Cloud Providers
		AWS
		Azure
		Google Cloud Platform
	Kubernetes
	Summary
Chapter 12. Operating Pulsar
	Apache BookKeeper Metrics
		Server Metrics
		Journal Metrics
		Storage Metrics
	Apache ZooKeeper Metrics
		Server Metrics
		Request Metrics
	Topic Metrics
	Consumer Metrics
	Pulsar Transaction Metrics
	Pulsar Function Metrics
	Advanced Operating Techniques
		Interceptors and Tracing
		Pulsar SQL Metrics
	Metrics Forwarding
		Dashboards
	Summary
Chapter 13. The Future
	Programming Language Support
		Extension Interface
		Enhancements to Pulsar Functions
		Architectural Simplification/Expansion
		Messaging Platform Bridges
	Summary
Appendix A. Pulsar Admin API
	Use Cases
	Examples
		Creating a Partitioned Topic
		Deleting a Partitioned Topic
		Creating a Namespace with Specific Policies
		Deleting a Namespace
	Summary
Appendix B. Pulsar Admin CLI
	CLI API
	Examples
		Creating a Partitioned Topic
		Creating a Pulsar IO Source
		Creating a Pulsar IO Sink
		Uploading a Schema
		Deleting a Schema
		Creating a Namespace
		Deleting a Namespace
	Summary
Appendix C. Geo-Replication
	Synchronous Replication
	Asynchronous Replication
	Replication Patterns
		Mesh
		Aggregation
		Standby
		Admin- and Producer-Level Control
	Summary
Appendix D. Security, Authentication, and Authorization in Pulsar
	Encryption in Transit
	Encryption at Rest
	Authentication
	Authorization
	Summary
Index
About the Author
Colophon