Kafka in Action

Kafka in Action
brief contents
contents
foreword
preface
acknowledgments
about this book
	Who should read this book?
	How this book is organized: A roadmap
	About the code
	liveBook discussion forum
	Other online resources
about the authors
about the cover illustration
Part 1 Getting started
	1 Introduction to Kafka
		1.1 What is Kafka?
		1.2 Kafka usage
			1.2.1 Kafka for the developer
			1.2.2 Explaining Kafka to your manager
		1.3 Kafka myths
			1.3.1 Kafka only works with Hadoop®
			1.3.2 Kafka is the same as other message brokers
		1.4 Kafka in the real world
			1.4.1 Early examples
			1.4.2 Later examples
			1.4.3 When Kafka might not be the right fit
		1.5 Online resources to get started
		References
	2 Getting to know Kafka
		2.1 Producing and consuming a message
		2.2 What are brokers?
		2.3 Tour of Kafka
			2.3.1 Producers and consumers
			2.3.2 Topics overview
			2.3.3 ZooKeeper usage
			2.3.4 Kafka’s high-level architecture
			2.3.5 The commit log
		2.4 Various source code packages and what they do
			2.4.1 Kafka Streams
			2.4.2 Kafka Connect
			2.4.3 AdminClient package
			2.4.4 ksqlDB
		2.5 Confluent clients
		2.6 Stream processing and terminology
			2.6.1 Stream processing
			2.6.2 What exactly-once means
		References
Part 2 Applying Kafka
	3 Designing a Kafka project
		3.1 Designing a Kafka project
			3.1.1 Taking over an existing data architecture
			3.1.2 A first change
			3.1.3 Built-in features
			3.1.4 Data for our invoices
		3.2 Sensor event design
			3.2.1 Existing issues
			3.2.2 Why Kafka is the right fit
			3.2.3 Thought starters on our design
			3.2.4 User data requirements
			3.2.5 High-level plan for applying our questions
			3.2.6 Reviewing our blueprint
		3.3 Format of your data
			3.3.1 Plan for data
			3.3.2 Dependency setup
		References
	4 Producers: Sourcing data
		4.1 An example
			4.1.1 Producer notes
		4.2 Producer options
			4.2.1 Configuring the broker list
			4.2.2 How to go fast (or go safer)
			4.2.3 Timestamps
		4.3 Generating code for our requirements
			4.3.1 Client and broker versions
		References
	5 Consumers: Unlocking data
		5.1 An example
			5.1.1 Consumer options
			5.1.2 Understanding our coordinates
		5.2 How consumers interact
		5.3 Tracking
			5.3.1 Group coordinator
			5.3.2 Partition assignment strategy
		5.4 Marking our place
		5.5 Reading from a compacted topic
		5.6 Retrieving code for our factory requirements
			5.6.1 Reading options
			5.6.2 Requirements
		References
	6 Brokers
		6.1 Introducing the broker
		6.2 Role of ZooKeeper
		6.3 Options at the broker level
			6.3.1 Kafka’s other logs: Application logs
			6.3.2 Server log
			6.3.3 Managing state
		6.4 Partition replica leaders and their role
			6.4.1 Losing data
		6.5 Peeking into Kafka
			6.5.1 Cluster maintenance
			6.5.2 Adding a broker
			6.5.3 Upgrading your cluster
			6.5.4 Upgrading your clients
			6.5.5 Backups
		6.6 A note on stateful systems
		6.7 Exercise
		References
	7 Topics and partitions
		7.1 Topics
			7.1.1 Topic-creation options
			7.1.2 Replication factors
		7.2 Partitions
			7.2.1 Partition location
			7.2.2 Viewing our logs
		7.3 Testing with EmbeddedKafkaCluster
			7.3.1 Using Kafka Testcontainers
		7.4 Topic compaction
		References
	8 Kafka storage
		8.1 How long to store data
		8.2 Data movement
			8.2.1 Keeping the original event
			8.2.2 Moving away from a batch mindset
		8.3 Tools
			8.3.1 Apache Flume
			8.3.2 Red Hat® Debezium™
			8.3.3 Secor
			8.3.4 Example use case for data storage
		8.4 Bringing data back into Kafka
			8.4.1 Tiered storage
		8.5 Architectures with Kafka
			8.5.1 Lambda architecture
			8.5.2 Kappa architecture
		8.6 Multiple cluster setups
			8.6.1 Scaling by adding clusters
		8.7 Cloud- and container-based storage options
			8.7.1 Kubernetes clusters
		References
	9 Management: Tools and logging
		9.1 Administration clients
			9.1.1 Administration in code with AdminClient
			9.1.2 kcat
			9.1.3 Confluent REST Proxy API
		9.2 Running Kafka as a systemd service
		9.3 Logging
			9.3.1 Kafka application logs
			9.3.2 ZooKeeper logs
		9.4 Firewalls
			9.4.1 Advertised listeners
		9.5 Metrics
			9.5.1 JMX console
		9.6 Tracing option
			9.6.1 Producer logic
			9.6.2 Consumer logic
			9.6.3 Overriding clients
		9.7 General monitoring tools
		References
Part 3 Going further
	10 Protecting Kafka
		10.1 Security basics
			10.1.1 Encryption with SSL
			10.1.2 SSL between brokers and clients
			10.1.3 SSL between brokers
		10.2 Kerberos and the Simple Authentication and Security Layer (SASL)
		10.3 Authorization in Kafka
			10.3.1 Access control lists (ACLs)
			10.3.2 Role-based access control (RBAC)
		10.4 ZooKeeper
			10.4.1 Kerberos setup
		10.5 Quotas
			10.5.1 Network bandwidth quota
			10.5.2 Request rate quotas
		10.6 Data at rest
			10.6.1 Managed options
		References
	11 Schema registry
		11.1 A proposed Kafka maturity model
			11.1.1 Level 0
			11.1.2 Level 1
			11.1.3 Level 2
			11.1.4 Level 3
		11.2 The Schema Registry
			11.2.1 Installing the Confluent Schema Registry
			11.2.2 Registry configuration
		11.3 Schema features
			11.3.1 REST API
			11.3.2 Client library
		11.4 Compatibility rules
			11.4.1 Validating schema modifications
		11.5 Alternative to a schema registry
		References
	12 Stream processing with Kafka Streams and ksqlDB
		12.1 Kafka Streams
			12.1.1 KStreams API DSL
			12.1.2 KTable API
			12.1.3 GlobalKTable API
			12.1.4 Processor API
			12.1.5 Kafka Streams setup
		12.2 ksqlDB: An event-streaming database
			12.2.1 Queries
			12.2.2 Local development
			12.2.3 ksqlDB architecture
		12.3 Going further
			12.3.1 Kafka Improvement Proposals (KIPs)
			12.3.2 Kafka projects you can explore
			12.3.3 Community Slack channel
		References
appendix A Installation
	A.1 Operating system (OS) requirements
	A.2 Kafka versions
	A.3 Installing Kafka on your local machine
		A.3.1 Prerequisite: Java
		A.3.2 Prerequisite: ZooKeeper
		A.3.3 Prerequisite: Kafka download
		A.3.4 Starting a ZooKeeper server
		A.3.5 Creating and configuring a cluster by hand
	A.4 Confluent Platform
		A.4.1 Confluent command line interface (CLI)
		A.4.2 Docker
	A.5 How to work with the book examples
		A.5.1 Building from the command line
	A.6 Troubleshooting
appendix B Client example
	B.1 Python Kafka clients
		B.1.1 Installing Python
		B.1.2 Python producer example
		B.1.3 Python consumer
	B.2 Client testing
		B.2.1 Unit testing in Java
		B.2.2 Kafka Testcontainers
	References
index
	A
	B
	C
	D
	E
	F
	G
	H
	I
	J
	K
	L
	M
	N
	O
	P
	Q
	R
	S
	T
	U
	V
	W
	Z
Kafka in Action - back