Genomics in the Azure Cloud: Scaling Your Bioinformatics Workloads Using Enterprise-Grade Solutions

Copyright
Table of Contents
Preface
	Who Should Read This Book
	How the Book Is Organized
	Software and Hardware Requirements
	Code Conventions and Downloads
	Conventions Used in This Book
	Using Code Examples
	O’Reilly Online Learning
	How to Contact Us
	Acknowledgments
Chapter 1. Essentials of Cloud Architecture
	Cloud Horsepower
		Considerations for the Cloud
		Three Benefits of the Cloud
	Types of Cloud Services
		Infrastructure Services
		Platform Services
		Software Services
	Azure Environment Organization
	Getting an Azure Account
	Welcome to the Azure Portal
		Setting Up a Resource Group
		Creating Resources
		Free Services
	Basics of the Bioinformatics Workflow
		Primary Analysis
		Secondary Analysis
		Tertiary Analysis
		Other Analyses
		Other File Formats
Chapter 2. Organizing Genomics Data with Data Lakes
	Organizing Your Genomics Data
		Going for Bronze, Silver, and Gold
		Letting Your Bioinformatics Workflow Dictate Your Data Lake Organization
		Planning for -omics and Non-omics Data Together
	Creating a Data Lake with Azure Storage
		Blob Storage Versus Data Lake Storage
	Balancing Costs Versus Performance in Data Storage
		The Goldilocks Method of Storage Tiers
		Genomics Data Lifecycle
	Managing Access Inside the Lake
		Role-Based Access Control
		Access-Control Lists
	Azure Open Datasets for Genomics
Chapter 3. Querying Variant Data in SQL
	Building a Genomics Data Warehouse
		Example: Lab Results
		Data Warehouse Architecture for Genomics
	Azure Synapse Analytics
		Creating an Azure Synapse Analytics Workspace
		Registering Services in Subscriptions
		Getting to Work in the Synapse Workspace
		Using Open Row Sets
		Creating External Tables
		Did Someone Say “Pool Party”?
	Connecting to More Data Sources
	Azure SQL DB
		Creating a Database in Azure SQL DB
	Relaxing at Your Genomics Data Lakehouse
		Efficient File Formats
Chapter 4. Orchestrating Data Movement and Transformation
	Creating Your Data Factory
	Getting Started with Data Movement
		Getting Data into Your Data Lake Using the Copy Data Tool
		Linking to NCBI’s FTP Server
		Transforming Data Using Data Flows
		Building and Triggering Pipelines for Automation
Chapter 5. Azure Databricks (and Apache Spark)
	Introduction to Apache Spark and Databricks
	Setting Up an Azure Databricks Workspace
		Connecting Databricks to Your Data Lake
	Processing Variant Data with the Glow Package
		Exploring DataFrames
	Automating Variant Data Processing
		Orchestrating a Databricks Notebook from Data Factory
		A Brief Interlude About Distributed File Formats
	Using Other Tools in Databricks
		Single-Node Bioinformatics Tools
		Koalas
		Hail
Chapter 6. Azure Machine Learning
	How to Scale Machine Learning Tasks
	Creating an Azure Machine Learning Workspace
	Training a Drug Sensitivity Model
		Creating a Compute Instance in Azure Machine Learning Studio
		Datastores and Datasets
		Experimenting with Cluster-Based Training
	Automating Model Training with AutoML
		Explainable Machine Learning
	Using Azure Machine Learning Not for Machine Learning
		Performing Alignment in a Notebook
		Custom Docker Images for Bioinformatics
Chapter 7. High-Performance Computing and Other Compute Services
	Bring Your Own Pipeline (BYOP)
		Why Azure for HPC?
	Azure Batch
		Scaling Workloads with Cromwell
	Azure CycleCloud
		Setting Up CycleCloud Clusters
	Microsoft Genomics
		Alignment and Variant Calling with the msgen Package
Chapter 8. Deployment, Security, Compliance, and Potpourri
	Automating the Deployment of Cloud Resources
		Dev, Staging, and Prod
		Lifting Your Deployment with ARMs and Biceps
	Security Planning
		Azure Active Directory
		Role-Based Access Controls and Access-Control Lists
	Compliance
		HIPAA, HITECH, and HITRUST
		Azure Blueprints
	Cost Considerations
		Azure Pricing Calculator
		Retail Pricing Versus Enterprise Agreements
		Budgeting Examples
	Quota Problems
		Please, Sir, Can I Have Some More (vCPUs)?
	Getting General Support
Conclusion
	Looking Backward
		Baby Azure
	What Else?
		Using Other Web-Based Bioinformatics Platforms
	Looking Forward
		Cheaper Sequencing = More Data
Index
About the Author
Colophon