Simplifying Data Engineering and Analytics with Delta: Create analytics-ready data that fuels artificial intelligence and business intelligence

Cover
Title Page
Copyright and Credits
Foreword
Contributors
Table of Contents
Preface
Section 1 – Introduction to Delta Lake and Data Engineering Principles
Chapter 1: Introduction to Data Engineering
	The motivation behind data engineering
		Use cases
		How big is big data?
		But isn\'t ML and AI all the rage today?
	Understanding the role of data personas
	Big data ecosystem
		What characterizes big data?
		Classifying data
		Reaping value from data
		Top challenges of big data systems
	Evolution of data systems
		Rise of cloud data platforms
		SQL and NoSQL systems
		OLTP and OLAP systems
	Distributed computing
		SMP and MPP computing
		Parallel and distributed computing
	Business justification for tech spending
		Strategy for business transformation to use data as an asset
		Big data trends and best practices
	Summary
Chapter 2: Data Modeling and ETL
	Technical requirements
	What is data modeling and why should you care?
		Advantages of a data modeling exercise
		Stages of data modeling
		Data modeling approaches for different data stores
	Understanding metadata – data about data
		Data catalog
		Types of metadata
		Why is metadata management the nerve center of data?
	Moving and transforming data using ETL
		Scenarios to consider for building ETL pipelines
		Job orchestration
	How to choose the right data format
		Text format versus binary format
		Row versus column formats
		When to use which format
		Leveraging data compression
	Common big data design patterns
		Ingestion
		Transformations
		Persist
	Summary
	Further reading
Chapter 3: Delta – The Foundation Block for Big Data
	Technical requirements
	Motivation for Delta
		A case of too many is too little
		Data silos to data swamps
		Characteristics of curated data lakes
		DDL commands
		DML commands
		APPEND
	Demystifying Delta
		Format layout on disk
	The main features of Delta
		ACID transaction support
		Schema evolution
		Unifying batch and streaming workloads
		Time travel
		Performance
	Life with and without Delta
		Lakehouse
	Summary
Section 2 – End-to-End Process of Building Delta Pipelines
Chapter 4: Unifying Batch and Streaming with Delta
	Technical requirements
	Moving toward real-time systems
		Streaming concepts
		Lambda versus Kappa architectures
	Streaming ETL
		Extract – file-based versus event-based streaming
		Transforming – stream processing
		Loading – persisting the stream
	Handling streaming scenarios
		Joining with other static and dynamic datasets
		Recovering from failures
		Handling late-arriving data
		Stateless and stateful in-stream operations
	Trade-offs in designing streaming architectures
		Cost trade-offs
		Handling latency trade-offs
		Data reprocessing
		Multi-tenancy
		De-duplication
	Streaming best practices
	Summary
Chapter 5: Data Consolidation in Delta Lake
	Technical requirements
	Why consolidate disparate data types?
	Delta unifies all types of data
		Structured data
		Semi-structured data
		Unstructured data
	Avoiding patches of data darkness
		Addressing problems in flight status using Delta
		Augmenting domain knowledge constraints to quality
		Continuous quality monitoring
	Curating data in stages for analytics
		RDD, DataFrames, and datasets
		Spark transformations and actions
		Spark APIs and UDFs
	Ease of extending to existing and new use cases
		Delta Lake connectors
		Specialized Delta Lakes by industry
	Data governance
		GDPR and CCPA compliance
		Role-based data access
	Summary
Chapter 6: Solving Common Data Pattern Scenarios with Delta
	Technical requirements
	Understanding use case requirements
	Minimizing data movement with Delta time travel
	Delta cloning
	Handling CDC
		CDC
		Change Data Feed (CDF)
	Handling Slowly Changing Dimensions (SCD)
		SCD Type 1
		SCD Type 2
	Summary
Chapter 7: Delta for Data Warehouse Use Cases
	Technical requirements
	Choosing the right architecture
	Understanding what a data warehouse really solves
		Lacunas of data warehouses
	Discovering when a data lake does not suffice
	Addressing concurrency and latency requirements with Delta
	Visualizing data using BI reporting
		Can cubes be constructed with Delta?
	Analyzing tradeoffs in a push versus pull data flow
		Why is being open such a big deal?
	Considerations around data governance
	The rise of the lakehouse category
	Summary
Chapter 8: Handling Atypical Data Scenarios with Delta
	Technical requirements
	Emphasizing the importance of exploratory data analysis (EDA)
		From big data to good data
		Data profiling
		Statistical analysis
	Applying sampling techniques to address class imbalance
		How to detect and address imbalance
		Synthetic data generation to deal with data imbalance
	Addressing data skew
	Providing data anonymity
	Handling bias and variance in data
		Bias versus variance
		How do we detect bias and variance?
		How do we fix bias and variance?
	Compensating for missing and out-of-range data
	Monitoring data drift
	Summary
Chapter 9: Delta for Reproducible Machine Learning Pipelines
	Technical requirements
	Data science versus machine learning
	Challenges of ML development
	Formalizing the ML development process
		What is a model?
		What is MLOps?
		Aspirations of a modern ML platform
	The role of Delta in an ML pipeline
		Delta-backed feature store
		Delta-backed model training
		Delta-backed model inferencing
		Model monitoring with Delta
	From business problem to insight generation
	Summary
Chapter 10: Delta for Data Products and Services
	Technical requirements
	DaaS
	The need for data democratization
	Delta for unstructured data
		NLP data (text and audio)
		Image and video data
	Data mashups using Delta
		Data blending
		Data harmonization
		Federated query
	Facilitating data sharing with Delta
		Setting up Delta sharing
		Benefits of Delta sharing
		Data clean room
	Summary
Section 3 – Operationalizing and Productionalizing Delta Pipelines
Chapter 11: Operationalizing Data and ML Pipelines
	Technical requirements
	Why operationalize?
	Understanding and monitoring SLAs
	Scaling and high availability
	Planning for DR 
		How to decide on the correct DR strategy
		How Delta helps with DR
	Guaranteeing data quality
	Automation of CI/CD pipelines 
		Code under version control
		Infrastructure as Code (IaC)
		Unit and integration testing
	Data as code – An intelligent pipeline
	Summary
Chapter 12: Optimizing Cost and Performance with Delta
	Technical requirements
	Improving performance with common strategies
		Where to look and what to look for
	Optimizing with Delta
		Changing the data layout in storage
		Other platform optimizations
		Automation
	Is cost always inversely proportional to performance?
	Best practices for managing performance
	Summary
Chapter 13: Managing Your Data Journey
	Provisioning a multi-tenant infrastructure
	Data democratization via policies and processes
	Capacity planning
	Managing and monitoring
	Data sharing
	Data migration
	COE best practices
	Summary
Index
Other Books You May Enjoy