Designing Cloud Data Platforms

brief contents
contents
preface
acknowledgments
about this book
	Who should read this book
	How this book is organized: A roadmap
	About the code
	liveBook discussion forum
about the authors
about the cover illustration
Chapter 1: Introducing the data platform
	1.1 The trends behind the change from data warehouses to data platforms
	1.2 Data warehouses struggle with data variety, volume, and velocity
		1.2.1 Variety
		1.2.2 Volume
		1.2.3 Velocity
		1.2.4 All the V’s at once
	1.3 Data lakes to the rescue?
	1.4 Along came the cloud
	1.5 Cloud, data lakes, and data warehouses: The emergence of cloud data platforms
	1.6 Building blocks of a cloud data platform
		1.6.1 Ingestion layer
		1.6.2 Storage layer
		1.6.3 Processing layer
		1.6.4 Serving layer
	1.7 How the cloud data platform deals with the three V’s
		1.7.1 Variety
		1.7.2 Volume
		1.7.3 Velocity
		1.7.4 Two more V’s
	1.8 Common use cases
Chapter 2: Why a data platform and not just a data warehouse
	2.1 Cloud data platforms and cloud data warehouses: The practical aspects
		2.1.1 A closer look at the data sources
		2.1.2 An example cloud data warehouse–only architecture
		2.1.3 An example cloud data platform architecture
	2.2 Ingesting data
		2.2.1 Ingesting data directly into Azure Synapse
		2.2.2 Ingesting data into an Azure data platform
		2.2.3 Managing changes in upstream data sources
	2.3 Processing data
		2.3.1 Processing data in the warehouse
		2.3.2 Processing data in the data platform
	2.4 Accessing data
	2.5 Cloud cost considerations
	2.6 Exercise answers
Chapter 3: Getting bigger and leveraging the Big 3: Amazon, Microsoft Azure, and Google
	3.1 Cloud data platform layered architecture
		3.1.1 Data ingestion layer
		3.1.2 Fast and slow storage
		3.1.3 Processing layer
		3.1.4 Technical metadata layer
		3.1.5 The serving layer and data consumers
		3.1.6 Orchestration and ETL overlay layers
	3.2 The importance of layers in a data platform architecture
	3.3 Mapping cloud data platform layers to specific tools
		3.3.1 AWS
		3.3.2 Google Cloud
		3.3.3 Azure
	3.4 Open source and commercial alternatives
		3.4.1 Batch data ingestion
		3.4.2 Streaming data ingestion and real-time analytics
		3.4.3 Orchestration layer
	3.5 Exercise answers
Chapter 4: Getting data into the platform
	4.1 Databases, files, APIs, and streams
		4.1.1 Relational databases
		4.1.2 Files
		4.1.3 SaaS data via API
		4.1.4 Streams
	4.2 Ingesting data from relational databases
		4.2.1 Ingesting data from RDBMSs using a SQL interface
		4.2.2 Full-table ingestion
		4.2.3 Incremental table ingestion
		4.2.4 Change data capture (CDC)
		4.2.5 CDC vendors overview
		4.2.6 Data type conversion
		4.2.7 Ingesting data from NoSQL databases
		4.2.8 Capturing important metadata for RDBMS or NoSQL ingestion pipelines
	4.3 Ingesting data from files
		4.3.1 Tracking ingested files
		4.3.2 Capturing file ingestion metadata
	4.4 Ingesting data from streams
		4.4.1 Differences between batch and streaming ingestion
		4.4.2 Capturing streaming pipeline metadata
	4.5 Ingesting data from SaaS applications
		4.5.1 No standard approach to API design
		4.5.2 No standard way to deal with full vs. incremental data exports
		4.5.3 Resulting data is typically highly nested JSON
	4.6 Network and security considerations for data ingestion into the cloud
		4.6.1 Connecting other networks to your cloud data platform
	4.7 Exercise answers
Chapter 5: Organizing and processing data
	5.1 Processing as a separate layer in the data platform
	5.2 Data processing stages
	5.3 Organizing your cloud storage
		5.3.1 Cloud storage containers and folders
	5.4 Common data processing steps
		5.4.1 File format conversion
		5.4.2 Data deduplication
		5.4.3 Data quality checks
	5.5 Configurable pipelines
	5.6 Exercise answers
Chapter 6: Real-time data processing and analytics
	6.1 Real-time ingestion vs. real-time processing
	6.2 Use cases for real-time data processing
		6.2.1 Retail use case: Real-time ingestion
		6.2.2 Online gaming use case: Real-time ingestion and real-time processing
		6.2.3 Summary of real-time ingestion vs. real-time processing
	6.3 When should you use real-time ingestion and/or real-time processing?
	6.4 Organizing data for real-time use
		6.4.1 The anatomy of fast storage
		6.4.2 How does fast storage scale?
		6.4.3 Organizing data in the real-time storage
	6.5 Common data transformations in real time
		6.5.1 Causes of duplicates in real-time systems
		6.5.2 Deduplicating data in real-time systems
		6.5.3 Converting message formats in real-time pipelines
		6.5.4 Real-time data quality checks
		6.5.5 Combining batch and real-time data
	6.6 Cloud services for real-time data processing
		6.6.1 AWS real-time processing services
		6.6.2 Google Cloud real-time processing services
		6.6.3 Azure real-time processing services
	6.7 Exercise answers
Chapter 7: Metadata layer architecture
	7.1 What we mean by metadata
		7.1.1 Business metadata
		7.1.2 Data platform internal metadata or “pipeline metadata”
	7.2 Taking advantage of pipeline metadata
	7.3 Metadata model
		7.3.1 Metadata domains
	7.4 Metadata layer implementation options
		7.4.1 Metadata layer as a collection of configuration files
		7.4.2 Metadata database
		7.4.3 Metadata API
	7.5 Overview of existing solutions
		7.5.1 Cloud metadata services
		7.5.2 Open source metadata layer implementations
	7.6 Exercise answers
Chapter 8: Schema management
	8.1 Why schema management
		8.1.1 Schema changes in a traditional data warehouse architecture
		8.1.2 Schema-on-read approach
	8.2 Schema-management approaches
		8.2.1 Schema as a contract
		8.2.2 Schema management in the data platform
		8.2.3 Monitoring schema changes
	8.3 Schema Registry Implementation
		8.3.1 Apache Avro schemas
		8.3.2 Existing Schema Registry implementations
		8.3.3 Schema Registry as part of a Metadata layer
	8.4 Schema evolution scenarios
		8.4.1 Schema compatibility rules
		8.4.2 Schema evolution and data transformation pipelines
	8.5 Schema evolution and data warehouses
		8.5.1 Schema-management features of cloud data warehouses
	8.6 Exercise answers
Chapter 9: Data access and security
	9.1 Different types of data consumers
	9.2 Cloud data warehouses
		9.2.1 AWS Redshift
		9.2.2 Azure Synapse
		9.2.3 Google BigQuery
		9.2.4 Choosing the right data warehouse
	9.3 Application data access
		9.3.1 Cloud relational databases
		9.3.2 Cloud key/value data stores
		9.3.3 Full-text search services
		9.3.4 In-memory cache
	9.4 Machine learning on the data platform
		9.4.1 Machine learning model lifecycle on a cloud data platform
		9.4.2 ML cloud collaboration tools
	9.5 Business intelligence and reporting tools
		9.5.1 Traditional BI tools and cloud data platform integration
		9.5.2 Using Excel as a BI tool
		9.5.3 BI tools that are external to the cloud provider
	9.6 Data security
		9.6.1 Users, groups, and roles
		9.6.2 Credentials and configuration management
		9.6.3 Data encryption
		9.6.4 Network boundaries
	9.7 Exercise Answers
Chapter 10: Fueling business value with data platforms
	10.1 Why you need a data strategy
	10.2 The analytics maturity journey
		10.2.1 SEE: Getting insights from data
		10.2.2 PREDICT: Using data to predict what to do
		10.2.3 DO: Making your analytics actionable
		10.2.4 CREATE: Going beyond analytics into products
	10.3 The data platform: The engine that powers analytics maturity
	10.4 Platform project stoppers
		10.4.1 Time does indeed kill
		10.4.2 User adoption
		10.4.3 User trust and the need for data governance
		10.4.4 Operating in a platform silo
		10.4.5 The dollar dance
index
	A
	B
	C
	D
	E
	F
	G
	H
	I
	K
	L
	M
	N
	O
	P
	R
	S
	T
	U
	V
	W