Apache Airflow Best Practices

Cover
Title Page
Copyright
Dedication
Contributors
Table of Contents
Preface
Part 1: Apache Airflow: History, What, and Why
Chapter 1: Getting Started with Airflow 2.0
	What is data orchestration?
		Industry use cases
	Exploring Apache Airflow
		Apache Airflow 2.0
		Standout features of Apache Airflow
		A look ahead
	Core concepts of Airflow
		Why Airflow may not be right
		When to choose Airflow
		Zen of Python
		Idempotency
		Code as configuration
	Skills to use Apache Airflow effectively
	Summary
Chapter 2: Core Airflow Concepts
	Technical requirements
	DAGs
		Decorators and a DAG definition
		Scheduling with Apache Airflow and moving away from CRON
	Tasks
		Task operators
		The first task – defining the DAG and extract
		Defining the transform task
		Xcoms
		Defining the load task
		Setting the flow of tasks and dependencies
		Executing the DAG example
		Task groups
		Triggers
	Summary
Part 2: Airflow Basics
Chapter 3: Components of Airflow
	Technical requirements
	Overall architecture
	Executors
		Local Executors (Sequential and Local)
		Parallelism
		Celery Executor (Remote Executor)
		Kubernetes Executor (Remote Executor)
		Dask Executor (Remote Executor)
		Kubernetes Local Executor (Hybrid Executor)
	Scheduler
	Summary
Chapter 4: Basics of Airflow and DAG Authoring
	Technical requirements
	Designing a DAG
		DAG authoring example architecture development
		DAG example overview
		Initial workflow requirements
		Bringing our first Airflow DAG together
	Extracting images from the NASA API
		The NASA API
		Building an API request in Jupyter Notebook
	Automating your code with a DAG
		Writing your first DAG
		Instantiating a DAG object
		Defining default arguments
		Defining the first task
		What are operators?
		Defining the first task’s Python code
		Defining the second task
		Setting the task order
	Summary
Part 3: Common Use Cases
Chapter 5: Connecting to External Sources
	Technical requirements
	Connectors make Apache Airflow
		Computing outside of Airflow
		Where are these connections?
		Connections stored in the metadata database
		A quick note about secrets being added through the Airflow UI
		Creating Connections from the CLI
		Testing of Connections
		Using environment variables
		Airflow metadata database
		Secrets management service
		Secrets Cache
		How to test environment variables and secret store Connections
		Best practices
		Building an email or Slack alert
		Key considerations
		Airflow notification types
		Email notification
		Creating a Slack webhook
		Creating the Airflow Connection
		Let’s build an example DAG
	Summary
Chapter 6: Extending Functionality with UI Plugins
	Technical requirements
	Understanding Airflow UI plugins
	Creating a metrics dashboard plugin
		Step 1 – project structure
		Step 2 – view implementation
		Step 3 – metrics dashboard HTML template
		Step 4 – plugin implementation
	Summary
	References
Chapter 7: Writing and Distributing Custom Providers
	Technical requirements
	Structuring your provider
		General directory structure
	Authoring your provider
		Registering our provider
		Authoring our hook
		Authoring our operators
		Authoring our sensor
		Testing
		Functional examples
	Summary
Chapter 8: Orchestrating a Machine Learning Workflow
	Technical requirements
	Basics of a machine learning-based project
		Our recommendation system – movies for you
	Designing our DAG
	Implementing the DAG
		Determining whether data has changed
		Fetching data
		Pre-processing stage
		KNN feature creation
		Deep learning model training
		Promoting assets to production
	Summary
Chapter 9: Using Airflow as a Driving Service
	Technical requirements
	QA testing service
		Designing the system
		Choosing how to configure our workflows
		Defining our general DAG topology
		Creating our DAGs from our configurations
		Scheduling (and unscheduling) our DAGs
	Summary
Part 4: Scale with Your Deployed Instance
Chapter 10: Airflow Ops: Development and Deployment
	Technical requirements
	DAG deployments
		Bundling
		De-coupled DAG delivery
	Repository structures
		Mono-repo
		Multi-repo
		Connection and Variable management
		Environment variables
		Secrets backends
	Airflow deployment methods
		Kubernetes
		Virtual machines
		Service providers
		Localized development
		Virtual environments
		Docker Compose
		Cloud development environments
	Testing
		Testing environments
		Testing DAGs
		Testing providers
		Testing Airflow
	Summary
Chapter 11: Airflow Ops Best Practices: Observation and Monitoring
	Technical requirements
	Monitoring core Airflow components
		Scheduler
		Metadata database
		Triggerer
		Executors/workers
		Web server
	Monitoring your DAGs
		Logging
		Alerting
		SLA monitoring
		Performance profiling
	Summary
Chapter 12: Multi-Tenancy in Airflow
	Technical requirements
	When to choose multi-tenancy
	Component configuration
		The Celery Executor
		The Kubernetes executor
		The scheduler and triggerer
		DAGs
		Web UI
	Summary
Chapter 13: Migrating Airflow
	Technical requirements
	General management activities for a migration
		Inventory
		Sequence
		Migrate
		Monitor
	Technical approaches for migration
		Automating code migrations
		QA/testing design
	Planning a migration between Airflow environments
		Connections and variables
		DAGs
	Summary
Index
About PACKT
Other Books You May Enjoy