The Azure Data Lakehouse Toolkit: Building and Scaling Data Lakehouses on Azure with Delta Lake, Apache Spark, Databricks, Synapse Analytics, and Snowflake

Table of Contents
About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Part I: Getting Started
	Chapter 1: The Data Lakehouse Paradigm
		Background
		​Architecture
		Ingestion and Processing
			Data Factory
			Databricks
			Functions and Logic Apps
			Synapse Analytics Serverless Pools
			Stream Analytics
			Messaging Hubs
		Storing and Serving
			Delta Lake
			Synapse Analytics Dedicated SQL Pools
			Relational Database
				Purchasing Models (SQL DTU vs. vCore Database)
				Service Tiers
				Deployment Models
			Non-relational Databases
			Snowflake
		Consumption
			Analysis Services
			Power BI
			Power Apps
			Advanced Analytics
			Cognitive Services
			Machine Learning
		Continuous Integration, Deployment, and Governance
			DevOps
			Purview
		Summary
Part II: Data Platforms
	Chapter 2: Snowflake
		Architecture
		Cost
		Security
			Azure Key Vault
			Azure Private Link
		Applications
		Replication and Failover
		Data Integration with Azure
			Data Lake Storage Gen2
			Real-Time Data Loading with ADLS gen2
			Data Factory
			Databricks
		Data Transformation
		Governance
			Column-Level Security
			Row-Level Security
			Access History
			Object Tagging
		Sharing
			Direct Share
			Data Marketplace
			Data Exchange
		Continuous Integration and Deployment
			Jenkins
			Azure DevOps
		Reporting
			Power BI
		Delta Lake, Machine Learning, and Constraints
			Delta Lake
			Machine Learning
			Constraints
		Summary
	Chapter 3: Databricks
		Workspaces
			Data Science and Engineering
			Machine Learning
			SQL
		Compute
		Storage
			Mount Data Lake Storage Gen2 Account
				Getting Started
				Create a Secret Scope
				Mount Data Lake Storage Gen2
				Read Data Lake Storage Gen2 from Databricks
		Delta Lake
		Reporting
		Real-Time Analytics
		Advanced Analytics
		Security and Governance
		Continuous Integration and Deployment
		Integration with Synapse Analytics
		Dynamic Data Encryption
		Data Profile
		Query Profile
		Constraints
		Identity
		Delta Live Tables Merge
		Summary
	Chapter 4: Synapse Analytics
		Workspaces
			Storage
				SQL Database (SQL Pools)
				Lake Database
				Integration Dataset
				External Datasets
			Development
			Integration
			Monitoring
			Management
		Reporting
		Continuous Integration and Deployment
		Real-Time Analytics
			Structured Streaming
			Synapse Link
		Advanced Analytics
		Security
		Governance
		Additional Features
			Delta Tables
			Machine Learning
			SQL Server Integration Services Integration Runtime (SSIS IR)
			Map Data Tool
			Data Sharing
			SQL Incremental
			Constraints
		Summary
Part III: Apache Spark ELT
	Chapter 5: Pipelines and Jobs
		Databricks
		Data Factory
			Mapping Data Flows
			HDInsight Spark Activity
			Scheduling and Monitoring
		Synapse Analytics Workspace
		Summary
	Chapter 6: Notebook Code
		PySpark
			Excel
			XML
			JSON
			ZIP
		Scala
		SQL
		Optimizing Performance
		Summary
Part IV: Delta Lake
	Chapter 7: Schema Evolution
		Schema Evolution Using Parquet Format
		Schema Evolution Using Delta Format
			Append
			Overwrite
		Summary
	Chapter 8: Change Data Feed
		Create Database and Tables
		Insert Data into Tables
		Change Data Capture
		Streaming Changes
		Summary
	Chapter 9: Clones
		Shallow Clones
		Deep Clones
		Summary
	Chapter 10: Live Tables
		Advantages of Delta Live Tables
		Create a Notebook
		Create and Run a Pipeline
		Schedule a Pipeline
		Explore Event Logs
		Summary
	Chapter 11: Sharing
		Architecture
		Share Data
		Access Data
		Sharing Data with Snowflake
		Summary
Part V: Optimizing Performance
	Chapter 12: Dynamic Partition Pruning
		Partitions
		Prerequisites
			DPP Commands
			Create Cluster
			Create Notebook and Mount Data Lake
			Create Fact Table
			Verify Fact Table Partitions
			Create Dimension Table
		Join Results Without DPP Filter
		Join Results with DPP Filter
		Summary
	Chapter 13: Z-Ordering and Data Skipping
		Prepare Data in Delta Lake
		Verify Data in Delta Lake
		Create Hive Table
		Run Optimize and Z-Order Commands
		Verify Data Skipping
		Summary
	Chapter 14: Adaptive Query Execution
		How It Works
			Prerequisites
		Comparing AQE Performance on Query with Joins
			Create Datasets
			Disable AQE
			Enable AQE
		Summary
	Chapter 15: Bloom Filter Index
		How a Bloom Filter Index Works
		Create a Cluster
		Create a Notebook and Insert Data
		Enable Bloom Filter Index
		Create Tables
		Create a Bloom Filter Index
		Optimize Table with Z-Order
		Verify Performance Improvements
		Summary
	Chapter 16: Hypers​pace​
		Prerequisites
		Create Parquet Files
		Run a Query Without an Index
		Import Hyperspace
		Read the Parquet Files to a Data Frame
		Create a Hyperspace Index
		Rerun the Query with Hyperspace Index
		Other Hyperspace Management APIs
		Summary
Part VI: Advanced Capabilities
	Chapter 17: Auto Loader
		Advanced Schema Evolution
		Prerequisites
			Generate Data from SQL Database
			Load Data to Azure Data Lake Storage Gen2
			Configure Resources in Azure Portal
			Configure Databricks
		Run Auto Loader in Databricks
			Configuration Properties
			Rescue Data
			Schema Hints
			Infer Column Types
			Add New Columns
		Managing Auto Loader Resources
			Read a Stream
			Write a Stream
			Explore Results
		Summary
	Chapter 18: Python Wheels
		Install Application Software
			Install Visual Studio Code and Python Extension
			Install Python
			Configure Python Interpreter Path for Visual Studio Code
			Verify Python Version in Visual Studio Code Terminal
		Set Up Wheel Directory Folders and Files
			Create Setup File
			Create Readme File
			Create License File
		Create Init File
			Create Package Function File
		Install Python Wheel Packages
			Install Wheel Package
			Install Check Wheel Package
		Create and Verify Wheel File
			Create Wheel File
			Check Wheel Contents
			Verify Wheel File
			Configure Databricks Environment
			Install Wheel to Databricks Library
		Create Databricks Notebook
			Mount Data Lake Folder
			Create Spark Database
			Verify Wheel Package
			Import Wheel Package
			Create Function Parameters
			Run Wheel Package Function
			Show Spark Tables
		Files in Databricks Repos
		Continuous Integration and Deployment
		Summary
	Chapter 19: Security and Controls
		Implement Cluster, Pool, and Jobs Access Control
		Implement Workspace Access Control
		Implement Other Access and Visibility Controls
			Table Access Control
			Personal Access Tokens
			Visibility Controls
		Example Row-Level Security Implementation
			Create New User Groups
			Load Sample Data
				Create Delta Tables
			Run Queries Using Row-Level Security
			Create Row-Level Secured Views and Grant Selective User Access
		Interaction with Azure Active Directory
		Summary
Index