3D IMAGING, ANALYSIS AND APPLICATIONS.

Preface
Contents
Contributors
1 Introduction
	1.1 Introduction
	1.2 A Historical Perspective on 3D Imaging
		1.2.1 Image Formation and Image Capture
		1.2.2 Binocular Perception of Depth
		1.2.3 Stereoscopic Displays
	1.3 The Development of Computer Vision
		1.3.1 Further Reading in Computer Vision
	1.4 Acquisition Techniques for 3D Imaging
		1.4.1 Passive 3D Imaging
		1.4.2 Active 3D Imaging
		1.4.3 Passive Stereo Versus Active Stereo Imaging
		1.4.4 Learned Depth Estimation
	1.5 Milestones in 3D Imaging and Shape Analysis
		1.5.1 Active 3D Imaging: An Early Optical Triangulation System
		1.5.2 Passive 3D Imaging: An Early Stereo System
		1.5.3 Passive 3D Imaging: The Essential Matrix
		1.5.4 Model Fitting: The RANSAC Approach to Feature Correspondence Analysis
		1.5.5 Active 3D Imaging: Advances in Scanning Geometries
		1.5.6 3D Registration: Rigid Transformation Estimation from 3D Correspondences
		1.5.7 3D Registration: Iterative Closest Points
		1.5.8 Passive 3D Imaging: The Fundamental Matrix and Camera Self-calibration
		1.5.9 3D Local Shape Descriptors: Spin Images
		1.5.10 Passive 3D Imaging: Flexible Camera Calibration
		1.5.11 3D Shape Matching: Heat Kernel Signatures
		1.5.12 Active 3D Imaging: Kinect
		1.5.13 Random-Forest Classifiers for Real-Time 3D Human Pose Recognition
		1.5.14 Convolutional–Recursive Deep Learning for 3D Object Classification
		1.5.15 VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition
		1.5.16 PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
		1.5.17 Dynamic Graph CNN for Learning on Point Clouds
		1.5.18 PointNetLK: Robust and Efficient Point Cloud Registration Using PointNet
	1.6 Book Roadmap
		1.6.1 Part I: 3D Image Acquisition, Representation, and Visualization
		1.6.2 Part II: 3D Shape Analysis and Inference
		1.6.3 Part III: 3D Imaging Applications
	References
Part I 3D Shape Acquisition, Representation  and Visualisation
2 Passive 3D Imaging
	2.1 Introduction
		2.1.1 Chapter Outline
	2.2 An Overview of Passive 3D Imaging Systems
		2.2.1 Multiple-View Approaches
		2.2.2 Single-View Approaches
	2.3 Camera Modeling
		2.3.1 Homogeneous Coordinates
		2.3.2 Perspective Projection Camera Model
		2.3.3 Radial Distortion
	2.4 Camera Calibration
		2.4.1 Estimation of a Scene-to-Image Planar Homography
		2.4.2 Basic Calibration
		2.4.3 Refined Calibration
		2.4.4 Calibration of a Stereo Rig
	2.5 Two-View Geometry
		2.5.1 Epipolar Geometry
		2.5.2 Essential and Fundamental Matrices
		2.5.3 The Fundamental Matrix for Pure Translation
		2.5.4 Computation of the Fundamental Matrix
		2.5.5 Two Views Separated by a Pure Rotation
		2.5.6 Two Views of a Planar Scene
	2.6 Rectification
		2.6.1 Rectification with Calibration Information
		2.6.2 Rectification Without Calibration Information
	2.7 Finding Correspondences
		2.7.1 Correlation-Based Methods
		2.7.2 Feature-Based Methods
	2.8 3D Reconstruction
		2.8.1 Stereo
		2.8.2 Structure from Motion
	2.9 Deep Learning for Passive 3D Imaging
		2.9.1 Deep Learning for Stereo Matching
		2.9.2 Deep Learning for Monocular Reconstruction
	2.10 Passive Multiple-View 3D Imaging Systems
		2.10.1 Stereo Cameras
		2.10.2 People Counting
		2.10.3 3D Modeling
		2.10.4 Visual SLAM
	2.11 Passive Versus Active 3D Imaging Systems
	2.12 Concluding Remarks
	2.13 Further Reading
	2.14 Software Resources
	2.15 Questions
	2.16 Exercises
	References
3 Active Triangulation 3D Imaging Systems for Industrial Inspection
	3.1 Introduction
		3.1.1 Historical Context
		3.1.2 Basic Measurement Principles
		3.1.3 Active Triangulation-Based Methods
		3.1.4 Chapter Outline
	3.2 Spot Scanners
		3.2.1 Spot Position Detection
	3.3 Stripe Scanners
		3.3.1 Camera Model
		3.3.2 Sheet-of-light Projector Model
		3.3.3 Triangulation for Stripe Scanners
	3.4 Area-Based Structured Light Systems
		3.4.1 Gray Code Methods
		3.4.2 Phase-Shift Methods
		3.4.3 Triangulation for a Structured Light System
	3.5 System Calibration
	3.6 Measurement Uncertainty
		3.6.1 Uncertainty Related to the Phase-Shift Algorithm
		3.6.2 Uncertainty Related to Intrinsic Parameters
		3.6.3 Uncertainty Related to Extrinsic Parameters
		3.6.4 Uncertainty as a Design Tool
	3.7 Experimental Characterization
		3.7.1 Low-Level Characterization
		3.7.2 System-Level Characterization
		3.7.3 Application-Based Characterization
	3.8 Selected Advanced Topics
		3.8.1 Thin Lens Equation
		3.8.2 Depth of Field
		3.8.3 Scheimpflug Condition
		3.8.4 Speckle and Uncertainty
		3.8.5 Laser Beam Thickness
		3.8.6 Artifacts Induced by the Laser Beam Thickness
		3.8.7 Lateral Resolution
		3.8.8 Interreflection
	3.9 Advanced Designs
		3.9.1 Auto-Synchronized Design
		3.9.2 Lateral-Synchronized Design
		3.9.3 Modified Lateral-Synchronized Design
		3.9.4 High-Resolution Fringe Projection Design
	3.10 Research Challenges
	3.11 Concluding Remarks
	3.12 Further Reading
	3.13 Questions
	3.14 Exercises
	References
4 Active Time-of-Flight 3D Imaging Systems for Medium-Range Applications
	4.1 Introduction
		4.1.1 Historical Context
		4.1.2 Basic Measurement Principles
		4.1.3 Time-of-Flight Methods
		4.1.4 Chapter Outline
	4.2 Point-Based Systems
		4.2.1 Pulse-Based Systems
		4.2.2 Phase-Based Systems
	4.3 Laser Trackers
		4.3.1 Good Practices
		4.3.2 Combining Laser Trackers with Other 3D Imaging Systems
	4.4 Multi-channel Systems
		4.4.1 Physical Scanning
		4.4.2 Digital Scanning
	4.5 Area-Based Systems
		4.5.1 Camera Model
		4.5.2 Phase-Based ToF for the Consumer Market
		4.5.3 Range-Gated Imaging
	4.6 Characterization of ToF System Performance
		4.6.1 Comparison to a Reference System
		4.6.2 Standards and Guidelines
		4.6.3 Other Research
		4.6.4 Finding Inconsistencies in Final 3D Models
	4.7 Experimental Results
		4.7.1 Terrestrial LiDAR Systems (TLS)
		4.7.2 Mobile LiDAR Systems (MLS)
	4.8 Sensor Fusion and Navigation
		4.8.1 Sensors for Mobile Applications
		4.8.2 Cloud-Based, High-Definition Map
		4.8.3 Absolute Positioning Systems
	4.9 ToF Versus Photogrammetry
		4.9.1 TLS Versus Architectural Photogrammetry
		4.9.2 LiDAR Versus Aerial Photogrammetry
		4.9.3 LT Versus Industrial Photogrammetry
	4.10 Research Challenges
	4.11 Concluding Remarks
	4.12 Further Reading
	4.13  Questions
	4.14  Exercises
	References
5 Consumer-Grade RGB-D Cameras
	5.1 Introduction
		5.1.1 Learning Objectives
		5.1.2 Historical Context
		5.1.3 Basic Measurement Principles
		5.1.4 Basic Design Considerations
		5.1.5 Chapter Outline
	5.2 Camera and Projector Models
		5.2.1 Pinhole Model
		5.2.2 Extrinsic Parameters
	5.3 Active Stereo Vision RGB-D Cameras
		5.3.1 Stereo Vision
		5.3.2 Active Stereo Vision
	5.4 Structured-Light RGB-D Cameras
		5.4.1 Temporal Encoding
	5.5 Phase-Based Time-of-Flight RGB-D Cameras
		5.5.1 Microsoft's Kinect II and Kinect Azure
	5.6 Texture Mapping
	5.7 Range Uncertainty and Lateral Resolution
		5.7.1 Triangulation
		5.7.2 Time-of-Flight
		5.7.3 Lateral Resolution
		5.7.4 Point Spacing
		5.7.5 Lateral Resolution Versus Range Uncertainty
	5.8 System Characterization and Calibration
		5.8.1 Metrological Approaches
		5.8.2 Application-Based Approaches
		5.8.3 Improving the Calibration of RGB-D Cameras
		5.8.4 Final Remarks Related to System Performance and Characterization
	5.9 Research Challenges
	5.10 Concluding Remarks
	5.11 Further Reading
	5.12 Questions
	5.13 Exercises
	References
6 3D Data Representation, Storage  and Processing
	6.1 Introduction
		6.1.1 Overview
	6.2 Representation of 3D Data
		6.2.1 Raw Data
		6.2.2 Surface Representations
		6.2.3 Solid-Based Representations
		6.2.4 Summary of Solid-Based Representations
	6.3 Polygon Meshes
		6.3.1 Mesh Storage
		6.3.2 Mesh Data Structures
	6.4 Subdivision Surfaces
		6.4.1 Doo–Sabin Scheme
		6.4.2 Catmull–Clark Scheme
		6.4.3 Loop Scheme
	6.5 Local Differential Properties
		6.5.1 Surface Normals
		6.5.2 Differential Coordinates and the Mesh Laplacian
	6.6 Compression and Levels of Detail
		6.6.1 Mesh Simplification
		6.6.2 QEM Simplification Summary
		6.6.3 Surface Simplification Results
	6.7 Current and Future Challenges
	6.8 Concluding Remarks
	6.9 Further Reading
	6.10 Questions and Exercises
		6.10.1 Questions
		6.10.2 Exercises
	References
Part II 3D Shape Analysis and Inference
7 3D Local Descriptors—from Handcrafted to Learned
	7.1 Introduction
	7.2 Background
	7.3 Related Works
		7.3.1 Handcrafted Local 3D Descriptors
		7.3.2 Learned Local 3D Descriptors
	7.4 Methods
		7.4.1 SHOT: Unique Signatures of Histograms for Local Surface Description
		7.4.2 Spin Images
		7.4.3 CGF: Compact Geometric Features
		7.4.4 PPF-FoldNet
	7.5 Dataset and Evaluation
	7.6 Results
	7.7 Open Challenges
	7.8 Further Reading
	7.9 Questions and Exercises
	7.10 Hands-On 3D Descriptors
	References
8 3D Shape Registration
	8.1 Introduction
		8.1.1 Chapter Outline
	8.2 Registration of Two Views
		8.2.1 Problem Statement
		8.2.2 The Iterative Closest Points (ICP) Algorithm
		8.2.3 ICP Extensions
	8.3 Advanced Techniques
		8.3.1 Registration of More Than Two Views
		8.3.2 Registration in Cluttered Scenes
		8.3.3 Deformable Registration
		8.3.4 Machine Learning Techniques
	8.4 Registration at Work
		8.4.1 Two-View Registration
		8.4.2 Multiple-View Registration
	8.5 Case Study 1: Pairwise Alignment with Outlier Rejection
	8.6 Case Study 2: ICP with Levenberg–Marquardt
		8.6.1 The LM-ICP Method
		8.6.2 Computing the Derivatives
		8.6.3 The Case of Quaternions
		8.6.4 Summary of the LM-ICP Algorithm
		8.6.5 Results and Discussion
	8.7 Case Study 3: Deformable ICP with Levenberg–Marquardt
		8.7.1 Surface Representation
		8.7.2 Cost Function
		8.7.3 Minimization Procedure
		8.7.4 Summary of the Algorithm
		8.7.5 Experiments
	8.8 Case Study 4: Computer-Aided Laparoscopy by Preoperative Data Registration
		8.8.1 Context
		8.8.2 Problem Statement
		8.8.3 Registration
		8.8.4 Validation
	8.9 Challenges and Future Directions
	8.10 Conclusion
	8.11 Further Reading
	8.12 Questions
	8.13 Exercises
	References
9 3D Shape Matching for Retrieval and Recognition
	9.1 Introduction
		9.1.1 Retrieval and Recognition Evaluation
		9.1.2 Chapter Outline
	9.2 Literature Review
		9.2.1 Shape Retrieval
		9.2.2 Shape Recognition
		9.2.3 Shape Correspondences
	9.3 3D Shape Matching Techniques
		9.3.1 PANORAMA
		9.3.2 Spin Images for Object Recognition
		9.3.3 Functional Maps
		9.3.4 Shape Retrieval with Heat Kernel Signatures
	9.4 Main Challenges for Future Research
	9.5 Concluding Remarks
	9.6 Further Reading
	9.7 Questions and Exercises
		9.7.1 Questions
		9.7.2 Exercises
	References
10 3D Morphable Models: The Face,  Ear and Head
	10.1 Introduction
		10.1.1 Model Training Data
		10.1.2 The Analysis-by-synthesis Application
		10.1.3 Chapter Structure
	10.2 Historical Perspective
	10.3 Outline of a Classical 3DMM Construction Process
		10.3.1 Normalised Mesh Parameterisation
		10.3.2 Mesh Alignment
		10.3.3 Statistical Modelling
	10.4 3D Face and Head Datasets
	10.5 Facial Landmarking
	10.6 Correspondence Establishment
		10.6.1 Non-rigid ICP
		10.6.2 Global Correspondence Optimisation
		10.6.3 Coherent Point Drift
		10.6.4 Laplace–Beltrami Mesh Manipulation
		10.6.5 Parameterisation Methods
		10.6.6 Correspondence Establishment Summary
	10.7 Procrustes Alignment
	10.8 Statistical Modelling
		10.8.1 Principal Component Analysis
		10.8.2 Gaussian Process Morphable Model
		10.8.3 Statistical Modelling Using Autoencoders
	10.9 Existing 3DMM Construction Pipelines
		10.9.1 LSFM Pipeline
		10.9.2 Basel Open Framework
	10.10 3D Face Models
	10.11 3D Head Models
	10.12 3D Ear Models
	10.13 3D Facial Symmetry and Asymmetry
	10.14 3DMM Evaluation Criteria
		10.14.1 Compactness
		10.14.2 Generalisation
		10.14.3 Specificity
		10.14.4 A Comparison of 3DMM Construction Pipelines
	10.15 3DMM Construction Using Deep Learning
	10.16 Applications of 3DMMs
		10.16.1 Fitting a 3D Model to 2D Images
		10.16.2 Shape Reconstruction with Missing Data
		10.16.3 Age Regression
	10.17 Research Challenges
	10.18 Concluding Remarks
	10.19 Further Reading
	10.20 Questions
	10.21 Exercises
	References
11 Deep Learning on 3D Data
	11.1 Introduction
	11.2 Background
		11.2.1 Datasets
		11.2.2 Related Work
	11.3 Deep Learning on Regularly Structured 3D Data
		11.3.1 Multi-view CNN
		11.3.2 Volumetric CNN
		11.3.3 3D Volumetric CNN Versus Multi-view CNN
		11.3.4 Two New Volumetric Convolutional Neural Networks
		11.3.5 Experimental Results
		11.3.6 Further Discussion
	11.4 Deep Learning on Point Clouds
		11.4.1 Problem Statement
		11.4.2 Properties of Point Sets in mathbbRN
		11.4.3 PointNet Architecture
		11.4.4 Theoretical Analysis
		11.4.5 Experimental Results
	11.5 Deep Learning on Meshes
		11.5.1 Spatial Domain Graph CNN
		11.5.2 Spectral Domain Graph CNN
	11.6 Conclusion and Outlook
	11.7 Further Reading
	11.8 Questions
	11.9 Exercises
	References
Part III 3D Imaging Applications
12 3D Face Recognition
	12.1 Introduction
		12.1.1 Chapter Outline
	12.2 Verification and Identification
		12.2.1 Evaluation of Face Verification
		12.2.2 Evaluation of Face Identification
	12.3 Context: A Brief Overview of 2D Face Recognition
	12.4 3D Recognition Versus 2D Recognition
	12.5 3D Face Image Representation and Visualization
	12.6 3D Face Datasets
		12.6.1 FRGC V2 3D Face Dataset
		12.6.2 The Bosphorus Dataset
	12.7 Holistic Versus Local Feature-Based Methods for 3D Recognition
	12.8 Processing Stages in Classical 3D Face Recognition
		12.8.1 Face Detection and Segmentation
		12.8.2 Removal of Spikes
		12.8.3 Filling of Holes and Missing Data
		12.8.4 Removal of Noise
		12.8.5 Fiducial Point Localization and Pose Correction
		12.8.6 Spatial Resampling
		12.8.7 Feature Extraction on Facial Surfaces
		12.8.8 Classifiers for 3D Face Matching
	12.9 ICP-based 3D Face Recognition
		12.9.1 ICP Outline
		12.9.2 A Critical Discussion of ICP
		12.9.3 A Typical ICP-based 3D Face Recognition Implementation
		12.9.4 ICP Variants and Other Surface Matching Approaches
	12.10 PCA-based 3D Face Recognition
		12.10.1 PCA System Training
		12.10.2 PCA Training Using Singular Value Decomposition
		12.10.3 PCA Testing
		12.10.4 PCA Performance
	12.11 LDA-Based 3D Face Recognition
		12.11.1 Two-Class LDA
		12.11.2 LDA with More Than Two Classes
		12.11.3 LDA in High Dimensional 3D Face Spaces
		12.11.4 LDA Performance
	12.12 Normals and Curvature in 3D Face Recognition
		12.12.1 Computing Curvature on a 3D Face Scan
	12.13 A Selection of Classical Techniques in 3D Face Recognition
		12.13.1 Fusion of Multiple Region Classifiers
		12.13.2 Iterative Closest Normal Points
		12.13.3 3D Face Recognition Using Radial Curves
		12.13.4 A Registration-free Approach Using Matching of 3D Keypoint Descriptors
		12.13.5 GA-based Nasal Patch Selection for Expression-Robustness
	12.14 Deep Learning Approaches to 3D Face Recognition
		12.14.1 Deep 3D Face Identification by Transfer Learning
		12.14.2 Large-scale 3D Face Recognition, Trained from Scratch
	12.15 Research Challenges
	12.16 Concluding Remarks
	12.17 Further Reading
	12.18 Questions
	12.19 Exercises
	References
13 3D Digitization of Cultural Heritage
	13.1 Introduction
		13.1.1 Visualization and Interaction
		13.1.2 Remote Visit
		13.1.3 Study and Research
		13.1.4 Digital Restoration and Reconstruction
		13.1.5 Heritage Monitoring for Conservation
		13.1.6 Physical Replicas
		13.1.7 3D Archives of Cultural Heritage
		13.1.8 Information Systems
	13.2 Previous Work on 3D Digitization of Heritage
		13.2.1 3D Digitization of Tangible Cultural Heritage
		13.2.2 3D Digitization of Intangible Cultural Heritage
		13.2.3 3D Reconstruction of Damaged or No-Longer Extant Monuments
	13.3 Capture, Modeling, and Storage of Digitized Cultural Heritage
		13.3.1 Capturing 3D Data from Cultural Heritage Assets
		13.3.2 Modeling Cultural Heritage from Raw Data
		13.3.3 Creating a 3D Repository for Cultural Heritage
	13.4 Experimental Application
		13.4.1 3D Digitization of a Little Clay Sculpture with Photogrammetry
		13.4.2 Overview of the Software Tools Used
	13.5 Open Challenges
	13.6 Conclusion
	13.7 Further Reading
	13.8 Questions
	References
14 3D Phenotyping of Plants
	14.1 Introduction
	14.2 Related Work
		14.2.1 Organ Tracking
		14.2.2 Plant Health Monitoring
		14.2.3 3D Reconstruction
		14.2.4 Rhythmic Pattern Detection
		14.2.5 Structural Analysis
	14.3 Key Techniques
		14.3.1 Terminologies
		14.3.2 Automated Systems for 3D Phenotyping
		14.3.3 Multiple-View Alignment
		14.3.4 Organ Segmentation
	14.4 Main Challenges
	14.5 Conclusion
	14.6 Further Reading
	14.7 Exercises
	References
Index