Low-Power Computer Vision: Improve the Efficiency of Artificial Intelligence

Cover
Half Title
Title Page
Copyright Page
Contents
Foreword
Rebooting Computing and Low-Power Computer Vision
Editors
SECTION I: Introduction
	CHAPTER 1: Book Introduction
		1.1. ABOUT THE BOOK
		1.2. CHAPTER SUMMARIES
			1.2.1. History of Low-Power Computer Vision Challenge
			1.2.2. Survey on Energy-Efficient Deep Neural Networks for Computer Vision
			1.2.3. Hardware Design and Software Practices for Efficient Neural Network Inference
			1.2.4. Progressive Automatic Design of Search Space for One-Shot Neural Architecture
			1.2.5. Fast Adjustable Threshold for Uniform Neural Network Quantization
			1.2.6. Power-efficient Neural Network Scheduling on Heterogeneous system on chips (SoCs)
			1.2.7. Efficient Neural Architecture Search
			1.2.8. Design Methodology for Low-Power Image Recognition Systems Design
			1.2.9. Guided Design for Efficient On-device Object Detection Model
			1.2.10. Quantizing Neural Networks for Low-Power Computer Vision
			1.2.11. A Practical Guide to Designing Efficient Mobile Architectures
			1.2.12. A Survey of Quantization Methods for Efficient Neural Network Inference
	CHAPTER 2: History of Low-Power Computer Vision Challenge
		2.1. REBOOTING COMPUTING
		2.2. LOW-POWER IMAGE RECOGNITION CHALLENGE (LPIRC): 2015–2019
		2.3. LOW-POWER COMPUTER VISION CHALLENGE (LPCVC): 2020
		2.4. WINNERS
		2.5. ACKNOWLEDGMENTS
	CHAPTER 3: Survey on Energy-Efficient Deep Neural Networks for Computer Vision
		3.1. INTRODUCTION
		3.2. BACKGROUND
			3.2.1. Computation Intensity of Deep Neural Networks
			3.2.2. Low-Power Deep Neural Networks
		3.3. PARAMETER QUANTIZATION
		3.4. DEEP NEURAL NETWORK PRUNING
		3.5. DEEP NEURAL NETWORK LAYER AND FILTER COMPRESSION
		3.6. PARAMETER MATRIX DECOMPOSITION TECHNIQUES
		3.7. NEURAL ARCHITECTURE SEARCH
		3.8. KNOWLEDGE DISTILLATION
		3.9. ENERGY CONSUMPTION—ACCURACY TRADEOFF WITH DEEP NEURAL NETWORKS
		3.10. GUIDELINES FOR LOW-POWER COMPUTER VISION
			3.10.1. Relationship between Low-Power Computer Vision Techniques
			3.10.2. Deep Neural Network and Resolution Scaling
		3.11. EVALUATION METRICS
			3.11.1. Accuracy Measurements on Popular Datasets
			3.11.2. Memory Requirement and Number of Operations
			3.11.3. On-device Energy Consumption and Latency
		3.12. SUMMARY AND CONCLUSIONS
SECTION II: Competition Winners
	CHAPTER 4: Hardware Design and Software Practices for Efficient Neural Network Inference
		4.1. HARDWARE AND SOFTWARE DESIGN FRAMEWORK FOR EFFICIENT NEURAL NETWORK INFERENCE
			4.1.1. Introduction
			4.1.2. From Model to Instructions
		4.2. ISA-BASED CNN ACCELERATOR: ANGEL-EYE
			4.2.1. Hardware Architecture
			4.2.2. Compiler
			4.2.3. Runtime Workflow
			4.2.4. Extension Support of Upsampling Layers
			4.2.5. Evaluation
			4.2.6. Practice on DAC-SDC Low-Power Object Detection Challenge
		4.3. NEURAL NETWORK MODEL OPTIMIZATION
			4.3.1. Pruning and Quantization
				4.3.1.1. Network Pruning
				4.3.1.2. Network Quantization
				4.3.1.3. Evaluation and Practices
			4.3.2. Pruning with Hardware Cost Model
				4.3.2.1. Iterative Search-based Pruning Methods
				4.3.2.2. Local Programming-based Pruning and the Practice in LPCVC’19
			4.3.3. Architecture Search Framework
				4.3.3.1. Framework Design
				4.3.3.2. Case Study Using the aw nas Framework: Black-box Search Space Tuning for Hardware-aware NAS
		4.4. SUMMARY
	CHAPTER 5: Progressive Automatic Design of Search Space for One-Shot Neural Architecture Search
		5.1. ABSTRACT
		5.2. INTRODUCTION
		5.3. RELATED WORK
		5.4. METHOD
			5.4.1. Problem Formulation and Motivation
			5.4.2. Progressive Automatic Design of Search Space
		5.5. EXPERIMENTS
			5.5.1. Dataset and Implement Details
			5.5.2. Comparison with State-of-the-art Methods
			5.5.3. Automatically Designed Search Space
			5.5.4. Ablation Studies
		5.6. CONCLUSION
	CHAPTER 6: Fast Adjustable Threshold for Uniform Neural Network Quantization
		6.1. INTRODUCTION
		6.2. RELATED WORK
			6.2.1. Quantization with Knowledge Distillation
			6.2.2. Quantization without Fine-tuning
			6.2.3. Quantization with Training/Fine-tuning
		6.3. METHOD DESCRIPTION
			6.3.1. Quantization with Threshold Fine-tuning
				6.3.1.1. Differentiable Quantization Threshold
				6.3.1.2. Batch Normalization Folding
				6.3.1.3. Threshold Scale
				6.3.1.4. Training of Asymmetric Thresholds
				6.3.1.5. Vector Quantization
			6.3.2. Training on the Unlabeled Data
			6.3.3. Quantization of Depth-wise Separable Convolution
				6.3.3.1. Scaling the Weights for MobileNet-V2 (with ReLU6)
		6.4. EXPERIMENTS AND RESULTS
			6.4.1. Experiments Description
				6.4.1.1. Researched Architectures
				6.4.1.2. Training Procedure
			6.4.2. Results
		6.5. CONCLUSION
	CHAPTER 7: Power-efficient Neural Network Scheduling
		7.1. INTRODUCTION TO NEURAL NETWORK SCHEDULING ON HETEROGENEOUS SoCs
			7.1.1. Heterogeneous SoC
			7.1.2. Network Scheduling
		7.2. COARSE-GRAINED SCHEDULING FOR NEURAL NETWORK TASKS: A CASE STUDY OF CHAMPION SOLUTION IN LPIRC2016
			7.2.1. Introduction to the LPIRC2016. Mission and the Solutions
			7.2.2. Static Scheduling for the Image Recognition Task
			7.2.3. Manual Load Balancing for Pipelined Fast R-CNN
			7.2.4. The Result of Static Scheduling
		7.3. FINE-GRAINED NEURAL NETWORK SCHEDULING ON POWER-EFFICIENT PROCESSORS
			7.3.1. Network Scheduling on SUs: Compiler-Level Techniques
			7.3.2. Memory-Efficient Network Scheduling
			7.3.3. The Formulation of the Layer-Fusion Problem by Computational Graphs
			7.3.4. Cost Estimation of Fused Layer-Groups
			7.3.5. Hardware-Aware Network Fusion Algorithm (HaNF)
			7.3.6. Implementation of the Network Fusion Algorithm
			7.3.7. Evaluation of Memory Overhead
			7.3.8. Performance on Different Processors
		7.4. SCHEDULER-FRIENDLY NETWORK QUANTIZATIONS
			7.4.1. The Problem of Layer Pipelining between CPU and Integer SUs
			7.4.2. Introduction to Neural Network Quantization for Integer Neural Accelerators
			7.4.3. Related Work of Neural Network Quantization
			7.4.4. Linear Symmetric Quantization for Low-Precision Integer Hardware
			7.4.5. Making Full Use of the Pre-Trained Parameters
			7.4.6. Low-Precision Representation and Quantization Algorithm
			7.4.7. BN Layer Fusion of Quantized Networks
			7.4.8. Bias and Scaling Factor Quantization for Low-Precision Integer Operation
			7.4.9. Evaluation Results
		7.5. SUMMARY
	CHAPTER 8: Efficient Neural Network Architectures
		8.1. STANDARD CONVOLUTION LAYER
		8.2. EFFICIENT CONVOLUTION LAYERS
		8.3. MANUALLY DESIGNED EFFICIENT CNN MODELS
		8.4. NEURAL ARCHITECTURE SEARCH
		8.5. HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH
			8.5.1. Latency Prediction
			8.5.2. Specialized Models for Different Hardware
			8.5.3. Handling Many Platforms and Constraints
		8.6. CONCLUSION
	CHAPTER 9: Design Methodology for Low-Power Image Recognition Systems
		9.1. DESIGN METHODOLOGY USED IN LPIRC 2017
			9.1.1. Object Detection Networks
			9.1.2. Throughput Maximization by Pipelining
			9.1.3. Software Optimization Techniques
				9.1.3.1. Tucker Decomposition
				9.1.3.2. CPU Parallelization
				9.1.3.3. 16-bit Quantization
				9.1.3.4. Post Processing
		9.2. IMAGE RECOGNITION NETWORK EXPLORATION
			9.2.1. Single Stage Detectors
			9.2.2. Software Optimization Techniques
			9.2.3. Post Processing
			9.2.4. Network Exploration
			9.2.5. LPIRC 2018. Solution
		9.3. NETWORK PIPELINING FOR HETEROGENEOUS PROCESSOR SYSTEMS
			9.3.1. Network Pipelining Problem
			9.3.2. Network Pipelining Heuristic
			9.3.3. Software Framework for Network Pipelining
			9.3.4. Experimental Results
		9.4. CONCLUSION AND FUTURE WORK
	CHAPTER 10: Guided Design for Efficient On-device Object Detection Model
		10.1. INTRODUCTION
			10.1.1. LPIRC Track 1 in 2018. and 2019
			10.1.2. Three Awards for Amazon team
		10.2. BACKGROUND
		10.3. AWARD-WINNING METHODS
			10.3.1. Quantization Friendly Model
			10.3.2. Network Architecture Optimization
			10.3.3. Training Hyper-parameters
			10.3.4. Optimal Model Architecture
			10.3.5. Neural Architecture Search
			10.3.6. Dataset Filtering
			10.3.7. Non-maximum Suppression Threshold
			10.3.8. Combination
		10.4. CONCLUSION
SECTION III: Invited Articles
	CHAPTER 11: Quantizing Neural Networks
		11.1. INTRODUCTION
		11.2. QUANTIZATION FUNDAMENTALS
			11.2.1. Hardware Background
			11.2.2. Uniform Affine Quantization
				11.2.2.1. Symmetric Uniform Quantization
				11.2.2.2. Power-of-two Quantizer
				11.2.2.3. Quantization Granularity
			11.2.3. Quantization Simulation
				11.2.3.1. Batch Normalization Folding
				11.2.3.2. Activation Function Fusing
				11.2.3.3. Other Layers and Quantization
			11.2.4. Practical Considerations
				11.2.4.1. Symmetric vs. Asymmetric Quantization
				11.2.4.2. Per-tensor and Per-channel Quantization
		11.3. POST-TRAINING QUANTIZATION
			11.3.1. Quantization Range Setting
			11.3.2. Cross-Layer Equalization
			11.3.3. Bias Correction
			11.3.4. AdaRound
			11.3.5. Standard PTQ Pipeline
			11.3.6. Experiments
		11.4. QUANTIZATION-AWARE TRAINING
			11.4.1. Simulating Quantization for Backward Path
			11.4.2. Batch Normalization Folding and QAT
			11.4.3. Initialization for QAT
			11.4.4. Standard QAT Pipeline
			11.4.5. Experiments
		11.5. SUMMARY AND CONCLUSIONS
	CHAPTER 12: Building Efficient Mobile Architectures
		12.1. INTRODUCTION
		12.2. ARCHITECTURE PARAMETERIZATIONS
			12.2.1. Network Width Multiplier
			12.2.2. Input Resolution Multiplier
			12.2.3. Data and Internal Resolution
			12.2.4. Network Depth Multiplier
			12.2.5. Adjusting Multipliers for Multi-criteria Optimizations
		12.3. OPTIMIZING EARLY LAYERS
		12.4. OPTIMIZING THE FINAL LAYERS
			12.4.1. Adjusting the Resolution of the Final Spatial Layer
			12.4.2. Reducing the Size of the Embedding Layer
		12.5. ADJUSTING NON-LINEARITIES: H-SWISH AND H-SIGMOID
		12.6. PUTTING IT ALL TOGETHER
	CHAPTER 13: A Survey of Quantization Methods for Efficient Neural Network Inference
		13.1. INTRODUCTION
		13.2. GENERAL HISTORY OF QUANTIZATION
		13.3. BASIC CONCEPTS OF QUANTIZATION
			13.3.1. Problem Setup and Notations
			13.3.2. Uniform Quantization
			13.3.3. Symmetric and Asymmetric Quantization
			13.3.4. Range Calibration Algorithms: Static vs. Dynamic Quantization
			13.3.5. Quantization Granularity
			13.3.6. Non-Uniform Quantization
			13.3.7. Fine-tuning Methods
				13.3.7.1. Quantization-Aware Training
				13.3.7.2. Post-Training Quantization
				13.3.7.3. Zero-shot Quantization
			13.3.8. Stochastic Quantization
		13.4. ADVANCED CONCEPTS: QUANTIZATION BELOW 8 BITS
			13.4.1. Simulated and Integer-only Quantization
			13.4.2. Mixed-Precision Quantization
			13.4.3. Hardware Aware Quantization
			13.4.4. Distillation-Assisted Quantization
			13.4.5. Extreme Quantization
			13.4.6. Vector Quantization
		13.5. QUANTIZATION AND HARDWARE PROCESSORS
		13.6. FUTURE DIRECTIONS FOR RESEARCH IN QUANTIZATION
		13.7. SUMMARY AND CONCLUSIONS
Bibliography
Index