Data Mesh in Action Version 4

Data Mesh in Action MEAP V04
Copyright
Welcome
brief contents
Front matter
	preface
	acknowledgments
	about this book
		Who should read this book
		How to Navigate this Book
		How to Use this Book
	The Messflix Case Study
	About the authors
Chapter 1: The What and Why of Data Mesh
	1.1 Data Mesh 101
		Definition of a Data Mesh
	1.2 Why the Data Mesh?
		1.2.1 Alternatives
		1.2.2 Data Warehouses & Data Lakes Inside the Data Mesh
		1.2.3 Data Mesh benefits
		The business perspective
		The technology perspective
	1.3 Use Case: The Snow Shoveling Business
	1.4 Data Mesh principles
		1.4.1 Domain-oriented decentralized data ownership and architecture
		1.4.2 Data as a product
		When can we call data a product?
		Data Product as an autonomous component
		1.4.3 Federated computational governance
		Federalization of Data Governance
		Computational elements of Data Governance
		1.4.4 Self-serve data infrastructure as a platform
	1.5 Back to Snow Shoveling
	1.6 Socio-technical architecture
		1.6.1 Conway’s law
		1.6.2 Team Topologies
		1.6.3 Cognitive Load
	1.7 Data Mesh challenges
		1.7.1 Technological challenge
		1.7.2 Data Management challenge
		1.7.3 Organizational challenge
	1.8 Summary
Chapter 2: Is Data Mesh Right for You?
	2.1 Analyzing Data Mesh drivers
		2.1.1 Business drivers
		Business strategy
		Business case and its data needs complexity
		2.1.2 Organizational drivers
		Socio-technical complexity
		Data maturity
		Software engineering maturity (CI/CD, DevOps)
		2.1.3 Domain-data drivers
		Domain and data model complexity
		Data diversity
		Data volume
		2.1.4 Minor organizational drivers
		Data governance maturity
		Data-savvy software engineers
		Following Domain-Driven Design approach
		2.1.5 Is Data Mesh a good fit for me?
	2.2 Data Mesh alternatives and complementary solutions
		2.2.1 Enterprise Data Warehouse
		2.2.2 Data Lake
		2.2.3 Data Lakehouse
		2.2.4 Data Fabric
		2.2.5 Data Mesh vs. the rest of the world
	2.3 Understanding a Data Mesh implementation effort
		2.3.1 Data Mesh development cycle
		2.3.2 Development cycle in the shoveling example
		2.3.3 Enabling the team
		Ensuring data governance
		Facilitating Data Product development
		Facilitating platform development
		Facilitating governance
		2.3.4 Development cycle in detail
		Choose a business goal to enable with data
		Define the Data Products needed
		Develop Data Products
		Collect feedback about the platform and analyze common needs
		Establish common policies and improve governance
		Develop the platform
	2.4 Summary
Chapter 3: Kickstart your Data Mesh MVP in a month
	3.1 Getting the lay of the land
		3.1.1 Drawing a system landscape diagram
		3.1.2 Stakeholder analysis
	3.2 Identifying candidates for the MVP implementation team
		3.2.1 Choosing development teams
		Choosing the cooperation model
		3.2.2 Choosing a data governance team
		Why do you need a data governance team?
		Who do you need in a data governance team?
		How do you select a data governance team?
	3.3 Setting up the minimal governance
		3.3.1 Defining Data Mesh value statement(s)
		3.3.2 Defining data governance policies
		3.3.3 Federating data governance
	3.4 Developing minimal Data Products
		3.4.1 Identifying domain-oriented datasets
		Adding business benefit to the mix
		3.4.2 Choosing Data Product Owners
		3.4.3 Deciding on the minimum viable data product description
		3.4.4 Developing the simplest tools to expose your data
	3.5 Setting up the minimal platform
		3.5.1 Ensuring platform-forced governability
		3.5.2 Ensuring platform security
	3.6 Summary
Chapter 4: Domain ownership
	4.1 Capturing and analyzing domains
		4.1.1 Domain-Driven Design 101
		4.1.2 Invite the right people
		4.1.3 Choose the correct workshop technique
		Domain Storytelling
	4.2 Apply ownership using domain decomposition
		4.2.1 Domain, subdomain, and business capability
		4.2.2 Decompose domain using business capability modeling
		4.2.3 How are domains and business capabilities related to data?
		Domain, business capability, Domain Dataset, and Data Products
		Application to case study
		4.2.4 Assign responsibilities to the Data Product-owning team
		Data
		Data pipelines, software, and source code
		Quality, cleansing, and deduplication
		Enriching and aggregations
		4.2.5 Choose the right team to own data
	4.3 Apply ownership using data use cases
		4.3.1 Data use cases
		Use case description template
		4.3.2 Model and bounded context
		4.3.3 Set up boundaries of use case-driven Data Products
		Financial Analysis/Reporting Data Product
		Script recommendations Data Product
		Movie trends Data Product
		4.3.4 Choose the right team to own data
	4.4 Apply ownership using design heuristics
		4.4.1 What is heuristic?
		4.4.2 Using design heuristics
		4.4.3 Designing heuristics and possible boundaries
		Align Data Product with a domain/business capability
		Align Data Products with possible usage or use cases
		Align Data Products with personas of consumers
		Align Data Product with the source system
		Align Data Product with the consuming dashboard/visualization/business intelligence tool
		Build the Data Product as a registry for a core business entity
		Create Data Product for a coherent group of Domain Datasets
		Avoid Data Products with a canonical model of the whole enterprise
		Align with usage contract
		Align with organizational boundaries
	4.5 Final landscape: The mesh of interconnected Data Products
		4.5.1 Messflix Data Mesh
		4.5.2 Data Products form a mesh
		4.5.3 Is it already a Data Mesh?
	4.6 Summary
Chapter 5: Data as a Product
	5.1 Applying Product Thinking
		5.1.1 Product Thinking analysis
		Cost Statement Data Product
		Scripts Data Product
		Movie Popularity Data Product
		5.1.2 Data Product Canvas
		Movie Cast Data Product Canvas
		Movie Trends Data Product Canvas
	5.2 What is a Data Product?
		5.2.1 Data Product definition
		5.2.2 Product not project
		5.2.3 What can be a Data Product?
	5.3 Data Product ownership
		5.3.1 Data Product Owner
		5.3.2 Data Product Owner responsibilities
		5.3.3 An agile DevOps team as a base for Data Product Development Team
		5.3.4 Data Product Owner and Product Owner
	5.4 Conceptual architecture of a Data Product
		5.4.1 External architecture view
		Data Product ports
		Database-like storage
		Files
		(REST) API
		Streams
		Visualisations
		5.4.2 Internal architecture view
		Datasets
		Metadata
		Code
	5.5 Data Product fundamental characteristics
		5.5.1 Self-described Data Product
		5.5.2 Introduction to Metadata
		5.5.3 Metadata as code
		5.5.4 Data Product metadata
		5.5.5 Domain Data Set metadata
		5.5.6 Other kinds of metadata
	5.6 Additional Data Product characteristics: FAIR and immutability
		5.6.1 Findability
		5.6.2 Accessibility
		5.6.3 Interoperable
		5.6.4 Reusable
		5.6.5 Immutable
		Do Data Products always have to be immutable?
	5.7 Summary
Chapter 6: Federated Computational Governance
	6.1 Data Governance in a nutshell
	6.2 Benefits of Data Governance
		6.2.1 Business value perspective
		6.2.2 Data usability perspective
		6.2.3 Data control perspective
	6.3 Planning Data Governance outcomes
		6.3.1 Hierarchy of Data Governance outcomes
		6.3.2 Strategic level outcomes
		Defining Value Statements
		Data policies
		Technology stack-related policies
		6.3.3 Tactical level outcomes
		Detailing policies
		Assigning accountability and responsibility for data
		6.3.4 Implementation-level outcomes
		Domain governance
		Central platform governance
	6.4 Federating Data Governance
		Thinking of Data Governance in terms of “sliders”
		6.4.1 Extreme ends of data governance models
		Centralized Data Governance Model: Top-down decision flow
		Decentralized Data Governance Model: Bottom-up decision flow
		6.4.2 Federated Data Governance model
		Decision space focus
		Governance structure focus
		People focus
		Messflix Data Governance Council
		Messflix Data Governance Steering Committee
		Messflix Central Platform Team
		Messflix Data Product Owner
		6.4.3 Setting-up Governance Team operations
	6.5 Making Data Governance computational
		6.5.1 Making policies Computational
		6.5.2 Automating policy checks
	6.6 Summary
Chapter 7: The self-serve data platform
	7.1 The MVP platform
		7.1.1 Platform definition
		Application to our platform
		7.1.2 Platform thinking
		Application to our platform
	7.2 Improvements with the concept ”X-as-a-Service”
		7.2.1 X-as-a-Service explained
		Workshop Tool-Building-as-a-Service
		X-as-a-Service continued
		X-as-a-Service observed
		7.2.2 X-as-a-Service applied
	7.3 Improvements with the concept of “Platform Architecture”
		7.3.1 Platform Architecture Explained
		7.3.2 Platform architecture applied
	7.4 Improvements for the data producers
	7.5 Summary
Chapter 8: Self-Serve Data Platforms Applications in Comparison
	8.1 Data Mesh on GCP
		8.1.1 Self-Serve Data Platform architecture
		8.1.2 Identifying the components of the Platform
		8.1.3 Identifying the components of the Data Product
		8.1.4 Workflows
		8.1.5 Variations
		8.1.6 Relation to Data Mesh ideas
		8.1.7 GCP architecture summary
	8.2 Data Mesh on AWS
		8.2.1 Self-Serve Data Platform architecture
		8.2.2 Identifying the components of the Platform
		8.2.3 Identifying the components of the Data Products
		8.2.4 Workflows
		8.2.5 Relation to Data Mesh ideas
		8.2.6 Variations
		8.2.7 AWS architecture summary
	8.3 Data Mesh on Databricks
		8.3.1 Self-Serve Data Platform architecture
		8.3.2 Identifying the components of the Platform
		8.3.3 Identifying the components of the Data Product
		8.3.4 Remarks
		8.3.5 Variations
		8.3.6 Databricks architecture summary
	8.4 Data Mesh on Kafka
		8.4.1 Self-Serve Data Platform architecture
		8.4.2 Identifying the components
		8.4.3 Remarks
		8.4.4 Kafka Architecture summary
	8.5 Summary
Chapter 9: Solution architecture design
	9.1 Capture and understand current architecture
		9.1.1 What is software architecture?
		9.1.2 How to document architecture: The C4 model
		9.1.3 Looking at the C4 model
	9.2 Architectural drivers of a Data Product design
		9.2.1 Architectural drivers
		Functional requirements
		Quality attributes
		Constraints
		Principles
		9.2.2 Capturing architectural drivers for a Data Product design
		Analyzing functional requirements
		Production Cost Statement Data Product requirements
		Choose the most important quality attributes and decide on the metric
		Look for constraints, principles, and quality attributes on the organizational level
	9.3 Design future architecture of a Data Product and related systems
		9.3.1 Design session
		9.3.2 File-based data product: Spreadsheet
		Cost Statement Data Product: Solution number 1
		Cost Statement Data Product: Solution number 2
		9.3.3 From monolith and microservice to a data product
		Cast Data Product
		Turning the database inside-out
		Scripts Data Product
		9.3.4 Exposing data for stream processing and batch processing
		Event stream as a Data Product port
		Data Lake and Data Products
	9.4 Summary