The Azure Data Lakehouse Toolkit: Building and Scaling Data Lakehouses on Azure with Delta Lake, Apache Spark, Databricks, Synapse Analytics, and Snowflake

Table of Contents
About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Chapter 1: The Data Lakehouse Paradigm
	Background
	​Architecture
	Ingestion and Processing
		Data Factory
		Databricks
		Functions and Logic Apps
		Synapse Analytics Serverless Pools
		Stream Analytics
		Messaging Hubs
	Storing and Serving
		Delta Lake
		Synapse Analytics Dedicated SQL Pools
		Relational Database
			Purchasing Models (SQL DTU vs. vCore Database)
			Service Tiers
			Deployment Models
		Non-relational Databases
		Snowflake
	Consumption
		Analysis Services
		Power BI
		Power Apps
		Advanced Analytics
		Cognitive Services
		Machine Learning
	Continuous Integration, Deployment, and Governance
		DevOps
		Purview
	Summary
Chapter 2: Snowflake
	Architecture
	Cost
	Security
		Azure Key Vault
		Azure Private Link
	Applications
	Replication and Failover
	Data Integration with Azure
		Data Lake Storage Gen2
		Real-Time Data Loading with ADLS gen2
		Data Factory
		Databricks
	Data Transformation
	Governance
		Column-Level Security
		Row-Level Security
		Access History
		Object Tagging
	Sharing
		Direct Share
		Data Marketplace
		Data Exchange
	Continuous Integration and Deployment
		Jenkins
		Azure DevOps
	Reporting
		Power BI
	Delta Lake, Machine Learning, and Constraints
		Delta Lake
		Machine Learning
		Constraints
	Summary
Chapter 3: Databricks
	Workspaces
		Data Science and Engineering
		Machine Learning
		SQL
	Compute
	Storage
		Mount Data Lake Storage Gen2 Account
			Getting Started
			Create a Secret Scope
			Mount Data Lake Storage Gen2
			Read Data Lake Storage Gen2 from Databricks
	Delta Lake
	Reporting
	Real-Time Analytics
	Advanced Analytics
	Security and Governance
	Continuous Integration and Deployment
	Integration with Synapse Analytics
	Dynamic Data Encryption
	Data Profile
	Query Profile
	Constraints
	Identity
	Delta Live Tables Merge
	Summary
Chapter 4: Synapse Analytics
	Workspaces
		Storage
			SQL Database (SQL Pools)
			Lake Database
			Integration Dataset
			External Datasets
		Development
		Integration
		Monitoring
		Management
	Reporting
	Continuous Integration and Deployment
	Real-Time Analytics
		Structured Streaming
		Synapse Link
	Advanced Analytics
	Security
	Governance
	Additional Features
		Delta Tables
		Machine Learning
		SQL Server Integration Services Integration Runtime (SSIS IR)
		Map Data Tool
		Data Sharing
		SQL Incremental
		Constraints
	Summary
Chapter 5: Pipelines and Jobs
	Databricks
	Data Factory
		Mapping Data Flows
		HDInsight Spark Activity
		Scheduling and Monitoring
	Synapse Analytics Workspace
	Summary
Chapter 6: Notebook Code
	PySpark
		Excel
		XML
		JSON
		ZIP
	Scala
	SQL
	Optimizing Performance
	Summary
Chapter 7: Schema Evolution
	Schema Evolution Using Parquet Format
	Schema Evolution Using Delta Format
		Append
		Overwrite
	Summary
Chapter 8: Change Data Feed
	Create Database and Tables
	Insert Data into Tables
	Change Data Capture
	Streaming Changes
	Summary
Chapter 9: Clones
	Shallow Clones
	Deep Clones
	Summary
Chapter 10: Live Tables
	Advantages of Delta Live Tables
	Create a Notebook
	Create and Run a Pipeline
	Schedule a Pipeline
	Explore Event Logs
	Summary
Chapter 11: Sharing
	Architecture
	Share Data
	Access Data
	Sharing Data with Snowflake
	Summary
Chapter 12: Dynamic Partition Pruning
	Partitions
	Prerequisites
		DPP Commands
		Create Cluster
		Create Notebook and Mount Data Lake
		Create Fact Table
		Verify Fact Table Partitions
		Create Dimension Table
	Join Results Without DPP Filter
	Join Results with DPP Filter
	Summary
Chapter 13: Z-Ordering and Data Skipping
	Prepare Data in Delta Lake
	Verify Data in Delta Lake
	Create Hive Table
	Run Optimize and Z-Order Commands
	Verify Data Skipping
	Summary
Chapter 14: Adaptive Query Execution
	How It Works
		Prerequisites
	Comparing AQE Performance on Query with Joins
		Create Datasets
		Disable AQE
		Enable AQE
	Summary
Chapter 15: Bloom Filter Index
	How a Bloom Filter Index Works
	Create a Cluster
	Create a Notebook and Insert Data
	Enable Bloom Filter Index
	Create Tables
	Create a Bloom Filter Index
	Optimize Table with Z-Order
	Verify Performance Improvements
	Summary
Chapter 16: Hypers​pace​
	Prerequisites
	Create Parquet Files
	Run a Query Without an Index
	Import Hyperspace
	Read the Parquet Files to a Data Frame
	Create a Hyperspace Index
	Rerun the Query with Hyperspace Index
	Other Hyperspace Management APIs
	Summary
Chapter 17: Auto Loader
	Advanced Schema Evolution
	Prerequisites
		Generate Data from SQL Database
		Load Data to Azure Data Lake Storage Gen2
		Configure Resources in Azure Portal
		Configure Databricks
	Run Auto Loader in Databricks
		Configuration Properties
		Rescue Data
		Schema Hints
		Infer Column Types
		Add New Columns
	Managing Auto Loader Resources
		Read a Stream
		Write a Stream
		Explore Results
	Summary
Chapter 18: Python Wheels
	Install Application Software
		Install Visual Studio Code and Python Extension
		Install Python
		Configure Python Interpreter Path for Visual Studio Code
		Verify Python Version in Visual Studio Code Terminal
	Set Up Wheel Directory Folders and Files
		Create Setup File
		Create Readme File
		Create License File
	Create Init File
		Create Package Function File
	Install Python Wheel Packages
		Install Wheel Package
		Install Check Wheel Package
	Create and Verify Wheel File
		Create Wheel File
		Check Wheel Contents
		Verify Wheel File
		Configure Databricks Environment
		Install Wheel to Databricks Library
	Create Databricks Notebook
		Mount Data Lake Folder
		Create Spark Database
		Verify Wheel Package
		Import Wheel Package
		Create Function Parameters
		Run Wheel Package Function
		Show Spark Tables
	Files in Databricks Repos
	Continuous Integration and Deployment
	Summary
Chapter 19: Security and Controls
	Implement Cluster, Pool, and Jobs Access Control
	Implement Workspace Access Control
	Implement Other Access and Visibility Controls
		Table Access Control
		Personal Access Tokens
		Visibility Controls
	Example Row-Level Security Implementation
		Create New User Groups
		Load Sample Data
			Create Delta Tables
		Run Queries Using Row-Level Security
		Create Row-Level Secured Views and Grant Selective User Access
	Interaction with Azure Active Directory
	Summary
Index