Towards Ubiquitous Low-power Image Processing Platforms

Preface
Contents
Contributors
Part I The Tulipp Reference Platform
	1 Challenges in the Realm of Embedded Real-Time ImageProcessing
		1.1 Introduction
		1.2 Image Processing Challenges in the Medical Domain
		1.3 Image Processing Challenges in the UAV Domain
		1.4 Image Processing Challenges in the Automotive Domain
		1.5 Looking into the Future: Neural Networks
		1.6 Conclusion
		References
	2 TRP: A Foundational Platform for High-Performance Low-Power Embedded Image Processing
		2.1 Introduction
		2.2 The Constraints of Embedded Image Processing
		2.3 Foundational Platforms
		2.4 The Tulipp Reference Platform and Its Instances
			2.4.1 The Tulipp Reference Platform (TRP)
			2.4.2 The Tulipp Reference Platform (TRP) Instances
				2.4.2.1 The Medical Instance
				2.4.2.2 The Space Instance
				2.4.2.3 The Automotive Instance
				2.4.2.4 The UAV Instance
				2.4.2.5 The Robotics Instance
		2.5 The Guidelines Concept
			2.5.1 Guideline Definition
			2.5.2 Guideline Generation Methodology
			2.5.3 Guideline Quality Assurance
		2.6 Conclusion
		References
Part II The Tulipp Starter Kit
	3 The Tulipp Hardware Platform
		3.1 Introduction
		3.2 Core Processor
			3.2.1 FPGA Concept
			3.2.2 SoCs: Zynq Architecture
			3.2.3 Xilinx Zynq Ultrascale+
		3.3 Modular Carrier: EMC2-DP
			3.3.1 SoM
			3.3.2 Power
			3.3.3 Configuration and Booting
			3.3.4 External Interfaces
		3.4 FMC Card: FM191-RU
			3.4.1 Power
			3.4.2 External Interfaces
		3.5 Mechanical Aspects
			3.5.1 Physical Characteristics
			3.5.2 Fastening
			3.5.3 Enclosure
		3.6 Conclusion
		References
	4 Operating Systems for Reconfigurable Computing: Concepts and Survey
		4.1 Introduction
		4.2 Operating System Concepts for Reconfigurable Systems
			4.2.1 Definitions
			4.2.2 RCOS Services
			4.2.3 Abstraction
			4.2.4 Virtualisation
		4.3 Operating System Implementations for Reconfigurable Architectures
			4.3.1 RC-Functionality in OS Kernel
			4.3.2 Operating System Extensions for RC-Functionality
			4.3.3 Closed Source RCOSes
			4.3.4 Hardware Acceleration of OS Modules
			4.3.5 RC-Frameworks
		4.4 Challenges and Trends
		4.5 Conclusion
		References
	5 STHEM: Productive Implementation of High-Performance Embedded Image Processing Applications
		5.1 Introduction
		5.2 The Generic Development Process (GDP)
		5.3 Realising the Generic Development Process
			5.3.1 Selecting the Implementation Approach
				5.3.1.1 Single-Language Approaches
				5.3.1.2 Multi-Language Approaches
			5.3.2 Selecting and Evaluating Performance Analysis Tools
		5.4 STHEM: The Tulipp Tool-Chain
		5.5 Conclusion
		References
	6 Lynsyn and LynsynLite: The STHEM Power MeasurementUnits
		6.1 Introduction
		6.2 Asynchronous Power Measurement Techniques
			6.2.1 Intrusive Periodic Sampling
			6.2.2 Non-intrusive Periodic Sampling
		6.3 The Lynsyn and LynsynLite PMUs
			6.3.1 Lynsyn
			6.3.2 LynsynLite
		6.4 Experimental Setup
		6.5 Hardware Characterisation
			6.5.1 Sampling Frequency
			6.5.2 Power Sensor
				6.5.2.1 Precision
				6.5.2.2 Accuracy
		6.6 System-Level Characterisation
			6.6.1 Performance Interference
			6.6.2 Power and Energy Interference
			6.6.3 Source Code Correlation
		6.7 Case Study
		6.8 Related Work
		6.9 Conclusion
		References
	7 Accelerated High-Level Synthesis Feature Detection for FPGAs Using HiFlipVX
		7.1 Introduction
		7.2 Related Work
		7.3 Implementation
			7.3.1 Overview
			7.3.2 Image Processing Functions
			7.3.3 FAST Corner Detector
			7.3.4 Canny Edge Detector
			7.3.5 OFB Feature Detector
		7.4 Evaluation
			7.4.1 System Setup and Tool Investigation
			7.4.2 Implementation and Synthesis Results
			7.4.3 Latency Results
		7.5 Conclusion
		References
Part III The Tulipp Starter Kit at Work
	8 UAV Use Case: Real-Time Obstacle Avoidance System for Unmanned Aerial Vehicles Based on Stereo Vision
		8.1 Introduction
		8.2 Implementing the Stereo Image Processing System
		8.3 Obstacle Detection and Collision Avoidance
		8.4 Evaluation
		8.5 Insights
		8.6 Conclusion
		References
	9 Robotics Use Case Scenarios
		9.1 Introduction
		9.2 VineScout
		9.3 SEMFIRE
			9.3.1 Use Case Description
			9.3.2 Human Intervention
			9.3.3 Challenges
			9.3.4 Functional and Technical Specification
			9.3.5 Artificial Perception Base System
			9.3.6 Computational Deployment on the Ranger Using the Tulipp Platform
		9.4 Conclusions and Future Work
		References
	10 Reducing the Radiation Dose by a Factor of 4 Thanksto Real-Time Processing on the Tulipp Platform
		10.1 Introduction to the Medical Use Case
		10.2 Medical X-Ray Video: The Need for Embedded Computation
		10.3 X-ray Noise Reduction Implementation
			10.3.1 Algorithm Insights
				10.3.1.1 Clean Image
				10.3.1.2 Pre-filtering
				10.3.1.3 Multiscale Edge and Contrast Filtering
				10.3.1.4 Post-Filtering
			10.3.2 Implementation and Optimisation Methodology
			10.3.3 Function Fusion to Increase Locality
			10.3.4 Memory Optimisation
			10.3.5 Code Linearisation
			10.3.6 Kernel Decomposition
			10.3.7 FPGA Implementation of the Filters
		10.4 Performance Optimisation Results
			10.4.1 Initial Profiling
			10.4.2 Gaussian Blur Optimisation
			10.4.3 Final Results
			10.4.4 Wrap Up
		10.5 Conclusion
		References
	11 Using the Tulipp Platform to Diagnose Cancer
		11.1 Introduction
		11.2 Application
		11.3 Conclusion
		References
	12 Space Use-Case: Onboard Satellite Image Classification
		12.1 Introduction
		12.2 Convolutional Neural Networks for Image Processing
			12.2.1 Multilayer Perceptrons
			12.2.2 Convolutional Topology
		12.3 Spiking Neural Networks for Embedded Applications
			12.3.1 Integrate and Fire Neuron Model
			12.3.2 Exporting Weights from FNN to SNN: Neural Network Conversion
		12.4 Algorithmic Solution
			12.4.1 An Hybrid Neural Network for Space Applications
			12.4.2 The Hybrid Neural Network Algorithm
		12.5 Hardware Solution
			12.5.1 Why Target the Tulipp Platform ?
			12.5.2 Our Hardware HNN Architecture
				12.5.2.1 Xilinx®  DPU IP
				12.5.2.2 Formal to Spiking Domain Interface
				12.5.2.3 SNN Accelerator
			12.5.3 Architecture Configuration Flow
		12.6 Results
			12.6.1 Resource Utilisation
			12.6.2 Power Consumption
			12.6.3 Performance
		12.7 Discussion
		12.8 Conclusion
		References
Part IV  The Tulipp Ecosystem
	13 The Tulipp Ecosystem
		13.1 Introduction
		13.2 The Ecosystem
		13.3 Conclusion
		Appendix: Selected Ecosystem Endorsements
		References
	14 Tulipp and ClickCV: How the Future Demands of Computer Vision Can Be Met Using FPGAs
		14.1 Introduction
		14.2 Trends in Computer Vision for Embedded Systems
			14.2.1 Deep Learning for Image Recognition
			14.2.2 Deep Learning for Other Applications
			14.2.3 Feature-Based and Optical Flow-Based vSLAM
			14.2.4 Image Stabilization
			14.2.5 Putting It All Together
		14.3 The Potential of FPGAs in Computer Vision
			14.3.1 The Early Impact of GPUs on Neural Networks
			14.3.2 The Advantages of Neural Networks on FPGAs
			14.3.3 The Advantages of FPGAs for Computer Vision Systems
		14.4 FPGA Embedded Computer Vision Toolsets
			14.4.1 Bringing C/C++ to FPGAs
			14.4.2 ClickCV: High Performance, Low Latency, Accessible Computer Vision for FPGAs
			14.4.3 FPGA Tools for Data Scientists
		14.5 Conclusion
		References
Index