Building Event-Driven Microservices: Leveraging Organizational Data at Scale

Cover
Copyright
Table of Contents
Preface
	Conventions Used in This Book
	O’Reilly Online Learning
	How to Contact Us
	Acknowledgments
Chapter 1. Why Event-Driven Microservices
	What Are Event-Driven Microservices?
	Introduction to Domain-Driven Design and Bounded Contexts
		Leveraging Domain Models and Bounded Contexts
		Aligning Bounded Contexts with Business Requirements
	Communication Structures
		Business Communication Structures
		Implementation Communication Structures
		Data Communication Structures
		Conway’s Law and Communication Structures
	Communication Structures in Traditional Computing
		Option 1: Make a New Service
		Option 2: Add It to the Existing Service
		Pros and Cons of Each Option
		The Team Scenario, Continued
		Conflicting Pressures
	Event-Driven Communication Structures
		Events Are the Basis of Communication
		Event Streams Provide the Single Source of Truth
		Consumers Perform Their Own Modeling and Querying
		Data Communication Is Improved Across the Organization
		Accessible Data Supports Business Communication Changes
	Asynchronous Event-Driven Microservices
		Example Team Using Event-Driven Microservices
	Synchronous Microservices
		Drawbacks of Synchronous Microservices
		Benefits of Synchronous Microservices
	Summary
Chapter 2. Event-Driven Microservice Fundamentals
	Building Topologies
		Microservice Topology
		Business Topology
	The Contents of an Event
	The Structure of an Event
		Unkeyed Event
		Entity Event
		Keyed Event
	Materializing State from Entity Events
	Event Data Definitions and Schemas
	Microservice Single Writer Principle
	Powering Microservices with the Event Broker
		Event Storage and Serving
		Additional Factors to Consider
	Event Brokers Versus Message Brokers
		Consuming from the Immutable Log
		Providing a Single Source of Truth
	Managing Microservices at Scale
		Putting Microservices into Containers
		Putting Microservices into Virtual Machines
		Managing Containers and Virtual Machines
	Paying the Microservice Tax
	Summary
Chapter 3. Communication and Data Contracts
	Event-Driven Data Contracts
		Using Explicit Schemas as Contracts
		Schema Definition Comments
		Full-Featured Schema Evolution
		Code Generator Support
		Breaking Schema Changes
	Selecting an Event Format
	Designing Events
		Tell the Truth, the Whole Truth, and Nothing but the Truth
		Use a Singular Event Definition per Stream
		Use the Narrowest Data Types
		Keep Events Single-Purpose
		Minimize the Size of Events
		Involve Prospective Consumers in the Event Design
		Avoid Events as Semaphores or Signals
	Summary
Chapter 4. Integrating Event-Driven Architectures with Existing Systems
	What Is Data Liberation?
		Compromises for Data Liberation
		Converting Liberated Data to Events
	Data Liberation Patterns
	Data Liberation Frameworks
	Liberating Data by Query
		Bulk Loading
		Incremental Timestamp Loading
		Autoincrementing ID Loading
		Custom Querying
		Incremental Updating
		Benefits of Query-Based Updating
		Drawbacks of Query-Based Updating
	Liberating Data Using Change-Data Capture Logs
		Benefits of Using Data Store Logs
		Drawbacks of Using Data Base Logs
	Liberating Data Using Outbox Tables
		Performance Considerations
		Isolating Internal Data Models
		Ensuring Schema Compatibility
		Capturing Change-Data Using Triggers
	Making Data Definition Changes to Data Sets Under Capture
		Handling After-the-Fact Data Definition Changes for the Query and CDC Log Patterns
		Handling Data Definition Changes for Change-Data Table Capture Patterns
	Sinking Event Data to Data Stores
	The Impacts of Sinking and Sourcing on a Business
	Summary
Chapter 5. Event-Driven Processing Basics
	Composing Stateless Topologies
		Transformations
		Branching and Merging Streams
	Repartitioning Event Streams
		Example: Repartitioning an Event Stream
	Copartitioning Event Streams
		Example: Copartitioning an Event Stream
	Assigning Partitions to a Consumer Instance
		Assigning Partitions with the Partition Assignor
		Assigning Copartitioned Partitions
		Partition Assignment Strategies
	Recovering from Stateless Processing Instance Failures
	Summary
Chapter 6. Deterministic Stream Processing
	Determinism with Event-Driven Workflows
	Timestamps
		Synchronizing Distributed Timestamps
		Processing with Timestamped Events
	Event Scheduling and Deterministic Processing
		Custom Event Schedulers
		Processing Based on Event Time, Processing Time, and Ingestion Time
		Timestamp Extraction by the Consumer
		Request-Response Calls to External Systems
	Watermarks
		Watermarks in Parallel Processing
	Stream Time
		Stream Time in Parallel Processing
	Out-of-Order and Late-Arriving Events
		Late Events with Watermarks and Stream Time
		Causes and Impacts of Out-of-Order Events
		Time-Sensitive Functions and Windowing
	Handling Late Events
	Reprocessing Versus Processing in Near-Real Time
	Intermittent Failures and Late Events
	Producer/Event Broker Connectivity Issues
	Summary and Further Reading
Chapter 7. Stateful Streaming
	State Stores and Materializing State from an Event Stream
	Recording State to a Changelog Event Stream
	Materializing State to an Internal State Store
		Materializing Global State
		Advantages of Using Internal State
		Disadvantages of Using Internal State
		Scaling and Recovery of Internal State
	Materializing State to an External State Store
		Advantages of External State
		Drawbacks of External State
		Scaling and Recovery with External State Stores
	Rebuilding Versus Migrating State Stores
		Rebuilding
		Migrating
	Transactions and Effectively Once Processing
		Example: Stock Accounting Service
		Effectively Once Processing with Client-Broker Transactions
		Effectively Once Processing Without Client-Broker Transactions
	Summary
Chapter 8. Building Workflows with Microservices
	The Choreography Pattern
		A Simple Event-Driven Choreography Example
		Creating and Modifying a Choreographed Workflow
		Monitoring a Choreographed Workflow
	The Orchestration Pattern
		A Simple Event-Driven Orchestration Example
		A Simple Direct-Call Orchestration Example
		Comparing Direct-Call and Event-Driven Orchestration
		Creating and Modifying an Orchestration Workflow
		Monitoring the Orchestration Workflow
	Distributed Transactions
		Choreographed Transactions: The Saga Pattern
		Orchestrated Transactions
	Compensation Workflows
	Summary
Chapter 9. Microservices Using Function-as-a-Service
	Designing Function-Based Solutions as Microservices
		Ensure Strict Membership to a Bounded Context
		Commit Offsets Only After Processing Has Completed
		Less Is More
	Choosing a FaaS Provider
	Building Microservices Out of Functions
	Cold Start and Warm Starts
	Starting Functions with Triggers
		Triggering Based on New Events: The Event-Stream Listener
		Triggering Based on Consumer Group Lag
		Triggering on a Schedule
		Triggering Using Webhooks
		Triggering on Resource Events
	Performing Business Work with Functions
	Maintaining State
	Functions Calling Other Functions
		Event-Driven Communication Pattern
		Direct-Call Pattern
	Termination and Shutdown
	Tuning Your Functions
		Allocating Sufficient Resources
		Batch Event-Processing Parameters
	Scaling Your FaaS Solutions
	Summary
Chapter 10. Basic Producer and Consumer Microservices
	Where Do BPCs Work Well?
		Integration with Existing and Legacy Systems
		Stateful Business Logic That Isn’t Reliant Upon Event Order
		When the Data Layer Does Much of the Work
		Independent Scaling of the Processing and Data Layer
	Hybrid BPC Applications with External Stream Processing
		Example: Using an External Stream-Processing Framework to Join Event Streams
	Summary
Chapter 11. Heavyweight Framework Microservices
	A Brief History of Heavyweight Frameworks
	The Inner Workings of Heavyweight Frameworks
	Benefits and Limitations
	Cluster Setup Options and Execution Modes
		Use a Hosted Service
		Build Your Own Full Cluster
		Create Clusters with CMS Integration
	Application Submission Modes
		Driver Mode
		Cluster Mode
	Handling State and Using Checkpoints
	Scaling Applications and Handling Event Stream Partitions
		Scaling an Application While It Is Running
		Scaling an Application by Restarting It
		Autoscaling Applications
	Recovering from Failures
	Multitenancy Considerations
	Languages and Syntax
	Choosing a Framework
	Example: Session Windowing of Clicks and Views
	Summary
Chapter 12. Lightweight Framework Microservices
	Benefits and Limitations
	Lightweight Processing
	Handling State and Using Changelogs
	Scaling Applications and Recovering from Failures
		Event Shuffling
		State Assignment
		State Replication and Hot Replicas
	Choosing a Lightweight Framework
		Apache Kafka Streams
		Apache Samza: Embedded Mode
	Languages and Syntax
	Stream-Table-Table Join: Enrichment Pattern
	Summary
Chapter 13. Integrating Event-Driven and Request-Response Microservices
	Handling External Events
		Autonomously Generated Events
		Reactively Generated Events
	Handling Autonomously Generated Analytical Events
	Integrating with Third-Party Request-Response APIs
	Processing and Serving Stateful Data
		Serving Real-Time Requests with Internal State Stores
		Serving Real-Time Requests with External State Stores
	Handling Requests Within an Event-Driven Workflow
		Processing Events for User Interfaces
	Micro-Frontends in Request-Response Applications
	The Benefits of Microfrontends
		Composition-Based Microservices
		Easy Alignment to Business Requirements
	Drawbacks of Microfrontends
		Potentially Inconsistent UI Elements and Styling
		Varying Microfrontend Performance
		Example: Experience Search and Review Application
	Summary
Chapter 14. Supportive Tooling
	Microservice-to-Team Assignment System
	Event Stream Creation and Modification
	Event Stream Metadata Tagging
	Quotas
	Schema Registry
	Schema Creation and Modification Notifications
	Offset Management
	Permissions and Access Control Lists for Event Streams
	State Management and Application Reset
	Consumer Offset Lag Monitoring
	Streamlined Microservice Creation Process
	Container Management Controls
	Cluster Creation and Management
		Programmatic Bringup of Event Brokers
		Programmatic Bringup of Compute Resources
		Cross-Cluster Event Data Replication
		Programmatic Bringup of Tooling
	Dependency Tracking and Topology Visualization
		Topology Example
	Summary
Chapter 15. Testing Event-Driven Microservices
	General Testing Principles
	Unit-Testing Topology Functions
		Stateless Functions
		Stateful Functions
	Testing the Topology
	Testing Schema Evolution and Compatibility
	Integration Testing of Event-Driven Microservices
	Local Integration Testing
		Create a Temporary Environment Within the Runtime of Your Test Code
		Create a Temporary Environment External to Your Test Code
		Integrate Hosted Services Using Mocking and Simulator Options
		Integrate Remote Services That Have No Local Options
	Full Remote Integration Testing
		Programmatically Create a Temporary Integration Testing Environment
		Testing Using a Shared Environment
		Testing Using the Production Environment
	Choosing Your Full-Remote Integration Testing Strategy
	Summary
Chapter 16. Deploying Event-Driven Microservices
	Principles of Microservice Deployment
	Architectural Components of Microservice Deployment
		Continuous Integration, Delivery, and Deployment Systems
		Container Management Systems and Commodity Hardware
	The Basic Full-Stop Deployment Pattern
	The Rolling Update Pattern
	The Breaking Schema Change Pattern
		Eventual Migration via Two Event Streams
		Synchronized Migration to the New Event Stream
	The Blue-Green Deployment Pattern
	Summary
Chapter 17. Conclusion
	Communication Layers
	Business Domains and Bounded Contexts
	Shareable Tools and Infrastructure
	Schematized Events
	Data Liberation and the Single Source of Truth
	Microservices
	Microservice Implementation Options
	Testing
	Deploying
	Final Words
Index
About the Author
Colophon