Deep Learning at Scale: At the Intersection of Hardware, Software, and Data

Cover
Copyright
Table of Contents
Preface
	Why Scaling Matters
	Who This Book Is For
	How This Book Is Organized
		Introduction
		Part I: Foundational Concepts of Deep Learning
		Part II: Distributed Training
		Part III: Extreme Scaling
	What You Need to Use This Book
	Setting Up Your Environment for Hands-on Exercises
	Using Code Examples
	Conventions Used in This Book
	O’Reilly Online Learning
	How to Contact Us
	Acknowledgments
Chapter 1. What Nature and History Have Taught Us About Scale
	The Philosophy of Scaling
		The General Law of Scaling
		History of Scaling Law
	Scalable Systems
		Nature as a Scalable System
		Our Visual System: A Biological Inspiration
	Artificial Intelligence: The Evolution of Learnable Systems
		It Takes Four to Tango
		Evolving Deep Learning Trends
	Scale in the Context of Deep Learning
		Six Development Considerations
		Scaling Considerations
	Summary
Part I. Foundational Concepts of Deep Learning
	Chapter 2. Deep Learning
		The Role of Data in Deep Learning
		Data Flow in Deep Learning
		Hands-On Exercise #1: Implementing Minimalistic Deep Learning
			Developing the Model
			The Embedded/Latent Space
			A Word of Caution
			The Learning Rate and Loss Landscape
			Scaling Consideration
			Profiling
		Hands-On Exercise #2: Getting Complex with PyTorch
			Model Input Data and Pipeline
			Model
			Auxiliary Utilities
			Putting It All Together
		Computation Graphs
		Inference
		Summary
	Chapter 3. The Computational Side of Deep Learning
		The Higgs Boson of the Digital World
			Floating-Point Numbers: The Faux Continuous Numbers
			Units of Data Measurement
			Data Storage Formats: The Trade-off of Latency and Throughput
		Computer Architecture
			The Birth of the Electromechanical Engine
			Memory and Persistence
			Computation and Memory Combined
		The Scaling Laws of Electronics
		Scaling Out Computation with Parallelization
			Threads Versus Processes: The Unit of Parallelization
			Hardware-Optimized Libraries for Acceleration
			Parallel Computer Architectures: Flynn’s and Duncan’s Taxonomies
		Accelerated Computing
			Popular Accelerated Devices for Deep Learning
			CUDA
			Accelerator Benchmarking
		Summary
	Chapter 4. Putting It All Together: Efficient Deep Learning
		Hands-On Exercise #1: GPT-2
			Exercise Objectives
			Model Architecture
			Implementation
			Running the Example
			Experiment Tracking
			Measuring to Understand the Limitations and Scale Out
			Transitioning from Language to Vision
		Hands-On Exercise #2: Vision Model with Convolution
			Model Architecture
			Running the Example
			Observations
		Graph Compilation Using PyTorch 2.0
			New Components of PyTorch 2.0
			Graph Execution in PyTorch 2.0
		Modeling Techniques to Scale Training on a Single Device
			Graph Compilation
			Reduced- and Mixed-Precision Training
			Memory Tricks for Efficiency
			Optimizer Efficiencies
			Model Input Pipeline Tricks
			Writing Custom Kernels in PyTorch 2.0 with Triton
		Summary
Part II. Distributed Training
	Chapter 5. Distributed Systems and Communications
		Distributed Systems
			The Eight Fallacies of Distributed Computing
			The Consistency, Availability, and Partition Tolerance (CAP) Theorem
			The Scaling Law of Distributed Systems
			Types of Distributed Systems
		Communication in Distributed Systems
			Communication Paradigm
			Communication Patterns
			Communication Technologies
			MPI
			Communication Initialization: Rendezvous
			Hands-On Exercise
		Scaling Compute Capacity
			Infrastructure Setup Options
			Provisioning of Accelerated Devices
			Workload Management
		Deep Learning Infrastructure Review
			Overview of Leading Deep Learning Clusters
			Similarities Between Today’s Most Powerful Systems
		Summary
	Chapter 6. Theoretical Foundations of Distributed Deep Learning
		Distributed Deep Learning
			Centralized DDL
			Decentralized DDL
		Dimensions of Scaling Distributed Deep Learning
			Partitioning Dimensions of Distributed Deep Learning
			Types of Distributed Deep Learning Techniques
			Choosing a Scaling Technique
		Measuring Scale
			End-to-End Metrics and Benchmarks
			Measuring Incrementally in a Reproducible Environment
		Summary
	Chapter 7. Data Parallelism
		Data Partitioning
			Implications of Data Sampling Strategies
			Working with Remote Datasets
		Introduction to Data Parallel Techniques
			Hands-On Exercise #1: Centralized Parameter Server Using RCP
			Hands-On Exercise #2: Centralized Gradient-Partitioned Joint Worker/Server Distributed Training
			Hands-On Exercise #3: Decentralized Asynchronous Distributed Training
		Centralized Synchronous Data Parallel Strategies
			Data Parallel (DP)
			Distributed Data Parallel (DDP)
			Zero Redundancy Optimizer–Powered Data Parallelism (ZeRO-DP)
			Fault-Tolerant Training
			Hands-On Exercise #4: Scene Parsing with DDP
			Hands-On Exercise #5: Distributed Sharded DDP (ZeRO)
		Building Efficient Pipelines
			Dataset Format
			Local Versus Remote
			Staging
			Threads Versus Processes: Scaling Your Pipelines
			Memory Tricks
			Data Augmentations: CPU Versus GPU
			JIT Acceleration
			Hands-On Exercise #6: Pipeline Efficiency with FFCV
		Summary
	Chapter 8. Scaling Beyond Data Parallelism: Model, Pipeline, Tensor, and Hybrid Parallelism
		Questions to Ask Before Scaling Vertically
		Theoretical Foundations of Vertical Scaling
			Revisiting the Dimensions of Scaling
			Operators’ Perspective of Parallelism Dimensions
			Data Flow and Communications in Vertical Scaling
		Basic Building Blocks for Scaling Beyond DP
			PyTorch Primitives for Vertical Scaling
			Working with Larger Models
			Distributed Checkpointing: Saving the Partitioned Model
		Summary
	Chapter 9. Gaining Practical Expertise with Scaling Across All Dimensions
		Hands-On Exercises: Model, Tensor, Pipeline, and Hybrid Parallelism
			The Dataset
			Hands-On Exercise #1: Baseline DeepFM
			Hands-On Exercise #2: Model Parallel DeepFM
			Hands-On Exercise #3: Pipeline Parallel DeepFM
			Hands-On Exercise #4: Pipeline Parallel DeepFM with RPC
			Hands-On Exercise #5: Tensor Parallel DeepFM
			Hands-On Exercise #6: Hybrid Parallel DeepFM
		Tools and Libraries for Vertical Scaling
			OneFlow
			FairScale
			DeepSpeed
			FSDP
			Overview and Comparison
			Hands-On Exercise #7: Automatic Vertical Scaling with DeepSpeed
			Observations
		Summary
Part III. Extreme Scaling
	Chapter 10. Data-Centric Scaling
		The Seven Vs of Data Through a Deep Learning Lens
		The Scaling Law of Data
		Data Quality
			Validity
			Variety
			Veracity
			Value and Volume
		The Data Engine and Continual Learning
			Volatility
			Velocity
		Summary
	Chapter 11. Scaling Experiments: Effective Planning and Management
		Model Development Is Iterative
		Planning for Experiments and Execution
			Simplify the Complex
			Fast Iteration for Fast Feedback
			Decoupled Iterations
			Feasibility Testing
			Developing and Scaling a Minimal Viable Solution
			Setting Up for Iterative Execution
		Techniques to Scale Your Experiments
			Accelerating Model Convergence
			Accelerating Learning Via Optimization and Automation
			Accelerating Learning by Increasing Expertise
			Learning with Scarce Supervision
		Hands-On Exercises
			Hands-On Exercise #1: Transfer Learning
			Hands-On Exercise #2: Hyperparameter Optimization
			Hands-On Exercise #3: Knowledge Distillation
			Hands-On Exercise #4: Mixture of Experts
			Hands-On Exercise #5: Contrastive Learning
			Hands-On Exercise #6: Meta-Learning
		Summary
	Chapter 12. Efficient Fine-Tuning of Large Models
		Review of Fine-Tuning Techniques
			Standard Fine Tuning
			Meta-Learning (Zero-/Few-Shot Learning)
			Adapter-Based Fine Tuning
			Low-Rank Tuning
		LoRA—Parameter-Efficient Fine Tuning
		Quantized LoRA (QLoRA)
		Hands-on Exercise: QLoRA-Based Fine Tuning
			Implementation Details
			Inference
			Exercise Summary
		Summary
	Chapter 13. Foundation Models
		What Are Foundation Models?
		The Evolution of Foundation Models
		Challenges Involved in Developing Foundation Models
			Measurement Complexity
			Deployment Challenges
			Propagation of Defects to All Downstream Models
			Legal and Ethical Considerations
			Ensuring Consistency and Coherency
		Multimodal Large Language Models
			Projection
			Gated Cross-Attention
			Query-Based Encoding
			Further Exploration
		Summary
Index
About the Author
Colophon