Azure Data Factory by Example: Practical Implementation for Data Engineers

Table of Contents
About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Chapter 1: Creating an Azure Data Factory Instance
	Get Started in Azure
		Create a Free Azure Account
		Explore the Azure Portal
	Create a Resource Group
	Create an Azure Data Factory
	Explore the Azure Data Factory User Experience
		Navigation Header Bar
		Navigation Sidebar
	Link to a Git Repository
		Create a Git Repository in Azure Repos
		Link the Data Factory to the Git Repository
	The ADF UX as a Web-Based IDE
	Chapter Review
		Key Concepts
		For SSIS Developers
Chapter 2: Your First Pipeline
	Work with Azure Storage
		Create an Azure Storage Account
		Explore Azure Storage
		Upload Sample Data
	Use the Copy Data Tool
	Explore Your Pipeline
		Linked Services
		Datasets
		Pipelines
		Activities
		Integration Runtimes
		Factory Resources in Git
	Debug Your Pipeline
		Run the Pipeline in Debug Mode
		Inspect Execution Results
	Chapter Review
		Key Concepts
		For SSIS Developers
Chapter 3: The Copy Data Activity
	Prepare an Azure SQL Database
		Create the Database
		Create Database Objects
	Import Structured Data into Azure SQL DB
		Create the Basic Pipeline
			Create the Database Linked Service and Dataset
			Create a DelimitedText File Dataset
			Create and Run the Pipeline
			Verify the Results
		Process Multiple Files
		Truncate Before Load
	Map Source and Sink Schemas
		Create a New Source Dataset
		Create a New Pipeline
		Configure Schema Mapping
	Import Semi-structured Data into Azure SQL DB
		Create a JSON File Dataset
		Create the Pipeline
		Configure Schema Mapping
		Set the Collection Reference
		The Effect of Schema Drift
		Understanding Type Conversion
	Transform JSON Files into Parquet
		Create a New JSON Dataset
		Create a Parquet Dataset
		Create and Run the Transformation Pipeline
	Performance Settings
		Data Integration Unit
		Degree of Copy Parallelism
	Chapter Review
		Key Concepts
		Azure Data Factory User Experience (ADF UX)
		For SSIS Developers
Chapter 4: Expressions
	Explore the Expression Builder
	Use System Variables
		Enable Storage of Audit Information
		Create a New Pipeline
		Add New Source Columns
		Run the Pipeline
	Access Activity Run Properties
		Create Database Objects
		Add Stored Procedure Activity
		Run the Pipeline
	Use the Lookup Activity
		Create Database Objects
		Configure the Lookup Activity
		Use Breakpoints
		Use the Lookup Value
		Update the Stored Procedure Activity
		Run the Pipeline
	User Variables
		Create a Variable
		Set a Variable
		Use the Variable
		Array Variables
	Concatenate Strings
		Infix Operators
		String Interpolation
		Escaping @
	Chapter Review
		Key Concepts
		For SSIS Developers
Chapter 5: Parameters
	Set Up an Azure Key Vault
		Create a Key Vault
		Create a Key Vault Secret
		Grant Access to the Key Vault
		Create a Key Vault ADF Linked Service
		Create a New Storage Account Linked Service
	Use Dataset Parameters
		Create a Parameterized Dataset
		Use the Parameterized Dataset
		Reuse the Parameterized Dataset
	Use Linked Service Parameters
		Create a Parameterized Linked Service
		Increase Dataset Reusability
		Use the New Dataset
		Why Parameterize Linked Services?
	Use Pipeline Parameters
		Create a Parameterized Pipeline
		Run the Parameterized Pipeline
		Use the Execute Pipeline Activity
		Parallel Execution
	Global Parameters
	Chapter Review
		Key Concepts
		For SSIS Developers
Chapter 6: Controlling Flow
	Create a Per-File Pipeline
	Use Activity Dependency Conditions
		Explore Dependency Condition Interactions
			Understand the Skipped Condition
			Understand the Failed Condition
			Combine Conditions
			Create Dependencies on Multiple Activities
			Understand the Completion Condition
		Understand Pipeline Outcome
	Raise Errors
	Use Conditional Activities
		Divert Error Rows
		Load Error Rows
			Create a New Sink Dataset
			Revise the Source Dataset
			Use the If Condition Activity
			Run the Pipeline
		Understand the Switch Activity
	Use Iteration Activities
		Use the Get Metadata Activity
		Use the ForEach Activity
		Ensure Parallelizability
		Understand the Until Activity
	Chapter Review
		Key Concepts
		For SSIS Developers
Chapter 7: Data Flows
	Build a Data Flow
		Enable Data Flow Debugging
		Add a Data Flow Transformation
		Use the Filter Transformation
		Use the Lookup Transformation
			Add a Lookup Data Stream
			Add the Lookup Transformation
		Use the Derived Column Transformation
		Use the Select Transformation
		Use the Sink Transformation
		Execute the Data Flow
			Create a Pipeline to Execute the Data Flow
			Inspect Execution Output
			Persist Loaded Data and Log Completion
	Maintain a Product Dimension
		Create a Dimension Table
		Create Supporting Datasets
		Build the Product Maintenance Data Flow
			Use Locals
			Use the Aggregate Transformation
			Use the Exists Transformation
		Execute the Dimension Data Flow
	Chapter Review
		Key Concepts
		For SSIS Developers
Chapter 8: Integration Runtimes
	Azure Integration Runtime
		Inspect the AutoResolveIntegrationRuntime
		Create a New Azure Integration Runtime
			Databricks Cluster TTL
			Controlling the Geography of Data Movement
		Use the New Azure Integration Runtime
			Identify the Copy Data Effective Integration Runtime
			Revise the Sink Integration Runtime
	Self-Hosted Integration Runtime
		Create a Shared Data Factory
		Create a Self-Hosted Integration Runtime
		Link to a Self-Hosted Integration Runtime
		Use the Self-Hosted Integration Runtime
			Create a Linked Service Using the Shared Self-Hosted IR
			Create a File System Dataset
			Copy Data Using the File System Dataset
	Azure-SSIS Integration Runtime
		Create an Azure-SSIS Integration Runtime
		Deploy SSIS Packages to the Azure-SSIS IR
		Run an SSIS Package in ADF
		Stop the Azure-SSIS IR
	Chapter Review
		Key Concepts
		For SSIS Developers
Chapter 9: Power Query in ADF
	Create a Power Query Mashup
	Explore the Power Query Editor
	Wrangle Data
	Run the Power Query Activity
	Chapter Review
Chapter 10: Publishing to ADF
	Publish to Your Factory Instance
		Trigger a Pipeline from the ADF UX
		Publish Factory Resources
		Inspect Published Pipeline Run Outcome
	Publish to Another Data Factory
		Prepare a Production Environment
			Create the Production Factory
			Grant Access to the Self-Hosted Integration Runtime
		Export ARM Template from Your Development Factory
		Import ARM Template into Your Production Factory
		Understand Deployment Parameters
	Automate Publishing to Another Factory
		Create a DevOps Service Connection
		Create an Azure DevOps Pipeline
			Create a YAML Pipeline File
			Create an Azure DevOps Pipeline Using the YAML File
			Add the Factory Deployment Task
		Trigger an Automatic Deployment
	Feature Branch Workflow
		Azure Data Factory Utilities
		Publish Resources as JSON
			Deploy ADF Pipelines Using PowerShell
			Resource Dependencies
	Chapter Review
Chapter 11: Triggers
	Use a Schedule Trigger
		Create a Schedule Trigger
		Reuse a Trigger
		Inspect Trigger Definitions
		Publish the Trigger
		Monitor Trigger Runs
		Deactivate the Trigger
		Advanced Recurrence Options
	Use an Event-Based Trigger
		Register the Event Grid Resource Provider
		Create an Event-Based Trigger
		Cause the Trigger to Run
		Trigger-Scoped System Variables
	Use a Tumbling Window Trigger
		Prepare Data
		Create a Windowed Copy Pipeline
		Create a Tumbling Window Trigger
		Monitor Trigger Runs
		Advanced Features
	Publishing Triggers Automatically
	Triggering Pipelines Programmatically
	Chapter Review
		Key Concepts
		For SSIS Developers
Chapter 12: Monitoring
	Generate Factory Activity
	Inspect Factory Logs
		Inspect Trigger Runs
		Inspect Pipeline Runs
		Add Metadata to the Log
			Add a Pipeline Annotation
			Add an Activity User Property
			Inspect Pipeline Annotations in the Log
			Inspect User Properties in the Log
	Inspect Factory Metrics
	Export Logs and Metrics
		Create a Log Analytics Workspace
		Configure Diagnostic Settings
		Inspect Logs in Blob Storage
	Use the Log Analytics Workspace
		Query Logs
		Use a Log Analytics Workbook
	Receive Alerts
		Configure Metric-Based Alerts
		Configure Log-Based Alerts
	Deactivate ADF Triggers
	Chapter Review
		Key Concepts
		For SSIS Developers
Index