Data Management at Scale: Best Practices for Enterprise Architecture

Cover
Copyright
Table of Contents
Foreword
Preface
	Navigating Through This Book
	Conventions Used in This Book
	O’Reilly Online Learning
	How to Contact Us
	Acknowledgments
Chapter 1. The Disruption of Data Management
	Data Management
	Analytics Is Fragmenting the Data Landscape
	Speed of Software Delivery Is Changing
	Networks Are Getting Faster
	Privacy and Security Concerns Are a Top Priority
	Operational and Transactional Systems Need to Be Integrated
	Data Monetization Requires an Ecosystem-to-Ecosystem Architecture
	Enterprises Are Saddled with Outdated Data Architectures
		Enterprise Data Warehouse and Business Intelligence
		Data Lake
		Centralized View
	Summary
		Scaled Architecture
Chapter 2. Introducing the Scaled Architecture: Organizing Data at Scale
	Universally Acknowledged Starting Points
		Each Application Has an Application Database
		Applications Are Specific and Have Unique Context
		Golden Source
		There’s No Escape from the Data Integration Dilemma
		Applications Play the Roles of Data Providers and Data Consumers
	Key Theoretical Considerations
		Object-Oriented Programming Principles
		Domain-Driven Design
		Business Architecture
	Communication and Integration Patterns
		Point-to-Point
		Silos
		Hub-Spoke Model
	Scaled Architecture
		Golden Sources and Domain Data Stores
		Data Delivery Contracts and Data Sharing Agreements
		Eliminating the Siloed Approach
		Domain-Driven Design on an Enterprise Scale
		Read-Optimized Data
		Data Layer as a Holistic Picture
		Metadata and the Target Operating Model
	Summary
Chapter 3. Managing Vast Amounts of Data: The Read-Only Data Stores Architecture
	Introducing the RDS Architecture
	Command and Query Responsibility Segregation
		What Is CQRS?
		CQRS at Scale
	Read-Only Data Store Components and Services
		Metadata
		Data Quality
		RDS Tiers
		Data Ingestion
		Integrating Commercial Off-the-Shelf Solutions
		Extracting Data from External APIs and SaaSs
		Historical Data Service
		Design Variations
		Data Replication
		Access Layer
		File Manipulation Service
		Delivery Notification Service
		De-Identification Service
		Distributed Orchestration
	Intelligent Consumption Services
	Populating RDSs on Demand
	RDS Direct Usage Considerations
	Summary
Chapter 4. Services and API Management: The API Architecture
	Introducing the API Architecture
	What Is Service-Oriented Architecture?
		Enterprise Application Integration
		Service Orchestration
		Service Choreography
		Public Services and Private Services
		Service Models and Canonical Data Models
		Similarities Between SOA and Enterprise Data Warehousing Architecture
	Modern View on SOA
		API Gateway
		Responsibility Model
		The New Role of the ESB
		Service Contracts
		Service Discovery
	Microservices
		The Role of the API Gateway Within Microservices
		Functions
		Service Mesh
		Microservices Boundaries
		Microservices Within the API Reference Architecture
	Ecosystem Communication
	API-Based Communication Channels
		GraphQL
		Backend for Frontend
	Metadata
	Using RDSs for Real-Time and Intensive Reads
	Summary
Chapter 5. Event and Response Management: The Streaming Architecture
	Introducing the Streaming Architecture
	The Asynchronous Event Model Makes the Difference
	What Do Event-Driven Architectures Look Like?
		Mediator Topology
		Broker Topology
		Event Processing Styles
	A Gentle Introduction to Apache Kafka
		Distributed Event Data
		Apache Kafka Features
	The Streaming Architecture
		Event Producers
		Event Consumers
		Event Platform
		Event Sourcing and Command Sourcing
		Governance Model
		Business Streams
		Streaming Consumption Patterns
		Event-Carried State Transfer
		Playing the Role of an RDS
		Using Streaming to Populate RDSs
		Controls and Policies for Guiding the Domains
	Streaming as the Operational Backbone
	Guarantees and Consistency
		Consistency Level
		“At Least Once, Exactly Once, and at Most Once” Processing
		Message Order
		Dead Letter Queue
		Streaming Interoperability
	Metadata for Governance and Self-Service Models
	Summary
Chapter 6. Connecting the Dots
	Recap of the Architectures
		RDS Architecture
		API Architecture
		Streaming Architecture
		Strengthening Patterns
	Enterprise Interoperability Standards
		Stable Data Endpoints
		Data Delivery Contracts
		Accessible and Addressable Data
		Crossing Network Principles
	Enterprise Data Standards
		Consumption-Optimization Principles
		Discoverability of Metadata
		Semantic Consistency
		Supplying the Corresponding Metadata
		Data Origination and Movements
	Reference Architecture
	Summary
Chapter 7. Sustainable Data Governance and Data Security
	Data Governance
		Organization: Data Governance Roles
		Processes: Data Governance Activities
		People: Trust and Ethical, Social, and Economic Considerations
		Technology: Golden Source, Ownership, and Application Administration
		Data: Golden Sources, Golden Datasets, and Classifications
	Data Security
		Current Siloed Approach
		Unified Data Security for Architectures
		Identity Providers
		Security Reference Architecture and Data Context Approach
		Security Process Flow
	Practical Guidance
		RDS Architecture
		API Architecture
		Streaming Architecture
		Intelligent Learning Engine
	Summary
Chapter 8. Turning Data into Value
	Consumption Patterns
		Using Read-Only Data Stores Directly
		Domain Data Stores
	Target Operating Model
	Data Professionals as a Target User Group
	Business Requirements
	Nonfunctional Requirements
	Building the Data Pipeline and Data Model
	Distributing Integrated Data
	Business Intelligence Capabilities
	Self-Service Capabilities
	Analytical Capabilities
		Standard Infrastructure for Automated Deployments
		Stateless Models
		Prescripted and Configured Workbenches
		Standardize on Model Integration Patterns
		Automation
		Model Metadata
	Advanced Analytics Reference Architecture
	Summary
Chapter 9. Mastering Enterprise Data Assets
	Demystifying Master Data Management
	Master Data Management Styles
	MDM Reference Architecture
		Designing a Master Data Management Solution
		MDM Distribution
		Master Identification Numbers
		Reference Data Versus Master Data
	Determining the Scope of Your Enterprise Data
	MDM and Data Quality as a Service
	Curated Data
		Metadata Exchange
		Integrated Views
		Reusable Components and Integration Logic
		Data Republishing
	Relation to Data Governance
	Summary
Chapter 10. Democratizing Data with Metadata
	Metadata Management
	Enterprise Metadata Model
	Enterprise Knowledge Graph
	Architectural Approaches for Metadata Management
		Metadata Interoperability
		Metadata Repositories
	Marketplace to Provide Rapid Access to Authorized Data
	Summary
Chapter 11. Conclusion
	Delivery Model
		Fully Decentralized Approach
		Partially Decentralized Approach
		Structuring Teams
		InnerSource Strategy
	Culture
	Technology Choices
	The Decline of Traditional Enterprise Architecture
		Blueprints and Diagrams
		Modern Skills
		Control and Governance
	Last Words
Glossary
Index
About the Author
Colophon