Efficient Processing of Deep Neural Networks

Preface
Acknowledgments
Understanding Deep Neural Networks
	Introduction
		Background on Deep Neural Networks
			Artificial Intelligence and Deep Neural Networks
			Neural Networks and Deep Neural Networks
		Training versus Inference
		Development History
		Applications of DNNs
		Embedded versus Cloud
	Overview of Deep Neural Networks
		Attributes of Connections Within a Layer
		Attributes of Connections Between Layers
		Popular Types of Layers in DNNs
			CONV Layer (Convolutional)
			FC Layer (Fully Connected)
			Nonlinearity
			Pooling and Unpooling
			Normalization
			Compound Layers
		Convolutional Neural Networks (CNNs)
			Popular CNN Models
		Other DNNs
		DNN Development Resources
			Frameworks
			Models
			Popular Datasets for Classification
			Datasets for Other Tasks
			Summary
Design of Hardware for Processing DNNs
	Key Metrics and Design Objectives
		Accuracy
		Throughput and Latency
		Energy Efficiency and Power Consumption
		Hardware Cost
		Flexibility
		Scalability
		Interplay Between Different Metrics
	Kernel Computation
		Matrix Multiplication with Toeplitz
		Tiling for Optimizing Performance
		Computation Transform Optimizations
			Gauss' Complex Multiplication Transform
			Strassen's Matrix Multiplication Transform
			Winograd Transform
			Fast Fourier Transform
			Selecting a Transform
		Summary
	Designing DNN Accelerators
		Evaluation Metrics and Design Objectives
		Key Properties of DNN to Leverage
		DNN Hardware Design Considerations
		Architectural Techniques for Exploiting Data Reuse
			Temporal Reuse
			Spatial Reuse
		Techniques to Reduce Reuse Distance
		Dataflows and Loop Nests
		Dataflow Taxonomy
			Weight Stationary (WS)
			Output Stationary (OS)
			Input Stationary (IS)
			Row Stationary (RS)
			Other Dataflows
			Dataflows for Cross-Layer Processing
		DNN Accelerator Buffer Management Strategies
			Implicit versus Explicit Orchestration
			Coupled versus Decoupled Orchestration
			Explicit Decoupled Data Orchestration (EDDO)
		Flexible NoC Design for DNN Accelerators
			Flexible Hierarchical Mesh Network
		Summary
	Operation Mapping on Specialized Hardware
		Mapping and Loop Nests
		Mappers and Compilers
		Mapper Organization
			Map Spaces and Iteration Spaces
			Mapper Search
			Mapper Models and Configuration Generation
		Analysis Framework for Energy Efficiency
			Input Data Access Energy Cost
			Partial Sum Accumulation Energy Cost
			Obtaining the Reuse Parameters
		Eyexam: Framework for Evaluating Performance
			Simple 1-D Convolution Example
			Apply Performance Analysis Framework to 1-D Example
		Tools for Map Space Exploration
Co-Design of DNN Hardware and Algorithms
	Reducing Precision
		Benefits of Reduce Precision
		Determining the Bit Width
			Quantization
			Standard Components of the Bit Width
		Mixed Precision: Different Precision for Different Data Types
		Varying Precision: Change Precision for Different Parts of the DNN
		Binary Nets
		Interplay Between Precision and Other Design Choices
		Summary of Design Considerations for Reducing Precision
	Exploiting Sparsity
		Sources of Sparsity
			Activation Sparsity
			Weight Sparsity
		Compression
			Tensor Terminology
			Classification of Tensor Representations
			Representation of Payloads
			Representation Optimizations
			Tensor Representation Notation
		Sparse Dataflow
			Exploiting Sparse Weights
			Exploiting Sparse Activations
			Exploiting Sparse Weights and Activations
			Exploiting Sparsity in FC Layers
			Summary of Sparse Dataflows
		Summary
	Designing Efficient DNN Models
		Manual Network Design
			Improving Efficiency of CONV Layers
			Improving Efficiency of FC Layers
			Improving Efficiency of Network Architecture After Training
		Neural Architecture Search
			Shrinking the Search Space
			Improving the Optimization Algorithm
			Accelerating the Performance Evaluation
			Example of Neural Architecture Search
		Knowledge Distillation
		Design Considerations for Efficient DNN Models
	Advanced Technologies
		Processing Near Memory
			Embedded High-Density Memories
			Stacked Memory (3-D Memory)
		Processing in Memory
			Non-Volatile Memories (NVM)
			Static Random Access Memories (SRAM)
			Dynamic Random Access Memories (DRAM)
			Design Challenges
		Processing in Sensor
		Processing in the Optical Domain
	Conclusion
	Bibliography
	Authors' Biographies