Springer Handbook of Augmented Reality

Foreword
Foreword
Foreword
Preface
Contents
About the Editors
Contributors
Part I Historical Developments
	1 Fundamentals of All the Realities: Virtual, Augmented, Mediated, Multimediated, and Beyond
		1.1 What Is (Augmented) Reality?
		1.2 Historical Background and Context
			1.2.1 A Confusing Mess of Different Realities: Virtual, Augmented, Mixed, and X-Reality
			1.2.2 Mediated Reality (XY-Reality)
			1.2.3 Deliberately Mediated Reality
			1.2.4 Unintentionally Mediated Reality
			1.2.5 The Mediated Reality (X,Y) Continuum
		1.3 Multimediated Reality
			1.3.1 Technologies for Sensory Attenuation
			1.3.2 Multimedia in Photographic Darkrooms
			1.3.3 Multimediated Reality Darkroom
			1.3.4 Comparison with Existing Measuring Instruments
		1.4 Multimediated Reality Is Multiscale, Multimodal, Multisensory, Multiveillant, and Multidimensional
			1.4.1 Multisensory Synthetic Synesthesia
			1.4.2 Multidimensional Multimediated Reality
		1.5 Multimediated Reality Continuum
			1.5.1 Multimediated Reality is ``*R'' (All R)
		1.6 Other Forms of Phenomenological Augmented Reality
			1.6.1 History of the SWIM
			1.6.2 SWIM Principle of Operation
			1.6.3 Visualizing Radio Waves with SWIM and Software-Defined Radio
			1.6.4 Electric Machines and SWIM
		1.7 Summary and Conclusions
		References
	2 History of Augmented Reality
		2.1 AR Basics: Operational Definition and Enabling Technologies
		2.2 AR Enabling Technologies
			2.2.1 Positional Tracking for AR
			2.2.2 Co-registration and Rendering of Visual Contents
			2.2.3 Visualization
		2.3 The Past of AR
		2.4 Current AR Technologies and Applications
			2.4.1 Main Application Fields
		2.5 Emerging Trends and Open Challenges for AR
		2.6 Conclusions
		References
Part II Principles and Fundamentals, Software Techniques, and Developments
	3 Principles of Object Tracking and Mapping
		3.1 Pose Estimation and Tracking Fundamentals
			3.1.1 Notation: Problem Formulation
				Coordinate System Transformations
				6DoF Pose of Rigid Bodies
			3.1.2 Cameras
				Perspective Camera Model
				Intrinsic Parameters
				Image Distortion
				Extrinsic Parameters
				From World to Pixel Coordinates
				Camera Calibration
				Spherical Camera Model
				Rolling Shutter Effects
			3.1.3 Inertial Sensors
				Gyroscopes
				Accelerometers
				Inertial Navigation
			3.1.4 Depth Sensors
			3.1.5 Geospatial Navigation Sensors
		3.2 Computer Vision Techniques
			3.2.1 Feature Matching
			3.2.2 Feature Tracking
				Optical Flow Estimation
			3.2.3 Pose Estimation from Feature Correspondences
				The DLT Algorithm
				Homography Pose Estimation
				Nonlinear Optimization
				Random Sample Consensus
			3.2.4 The Epipolar Constraint
			3.2.5 Line Tracking
			3.2.6 Direct Image Alignment
			3.2.7 Structure from Motion
				Triangulation
				Bundle Adjustment
			3.2.8 Depth Image Tracking/ICP
			3.2.9 Deep Learning in Computer Vision
				Convolutional Neural Networks
				Network Training
				Deep Learning in Tracking and Mapping
		3.3 Model-Based Tracking
			3.3.1 Marker Trackers
			3.3.2 3D Model Trackers
				Object Model Acquisition
				Feature Trackers
				Edge Trackers
				Direct Trackers
				Deep Learning-Based Trackers
				Hybrid Trackers
				Tracking by Detection/Pose Initialization
			3.3.3 Nonrigid and Articulated Objects
		3.4 SLAM
			3.4.1 Visual SLAM
				SLAM Architecture/Main Components
				Keypoint-Based SLAM
				Direct SLAM
			3.4.2 Visual-Inertial SLAM
			3.4.3 RGB-D SLAM
			3.4.4 Deep Learning for SLAM
				Towards Semantic SLAM
		3.5 Conclusion
		References
	4 3D Object and Hand Pose Estimation
		4.1 3D Object and Hand Pose Estimation for Augmented Reality
		4.2 Formalization
		4.3 Challenges of 3D Pose Estimation Using Computer Vision
		4.4 Early Approaches to 3D Pose Estimation and Their Limits
		4.5 Machine Learning and Deep Learning
		4.6 Datasets
			4.6.1 Datasets for Object Pose Estimation
			4.6.2 Datasets for Hand Pose Estimation
			4.6.3 Datasets for Object and Hand Pose Estimation
			4.6.4 Metrics
		4.7 Modern Approaches to 3D Object Pose Estimation
			4.7.1 BB8
			4.7.2 SSD-6D
			4.7.3 YOLO-6D
			4.7.4 PoseCNN
			4.7.5 DeepIM
			4.7.6 Augmented Autoencoders
			4.7.7 Robustness to Partial Occlusions: Oberweger's Method, Segmentation-Driven MeThod, PVNet
			4.7.8 DPOD and Pix2Pose
			4.7.9 Discussion
		4.8 3D Pose Estimation for Object Categories
		4.9 3D Hand Pose Estimation from Depth Maps
			4.9.1 DeepPrior++
			4.9.2 V2V-PoseNet
			4.9.3 A2J
			4.9.4 Discussion
		4.10 3D Hand Pose Estimation from an RGB Image
			4.10.1 Zimmerman's Method
			4.10.2 Iqbal's Method
			4.10.3 GANerated Hands
			4.10.4 3D Hand Shape and Pose Estimation: Ge's and Boukhayma's Methods
			4.10.5 Implementation in MediaPipe
			4.10.6 Manipulating Virtual Objects
		4.11 3D Object+Hand Pose Estimation
			4.11.1 ObMan and HOPS-Net
			4.11.2 H+O
			4.11.3 HOnnotate
		4.12 The Future of 3D Object and Hand Pose Estimation
		References
	5 Mixed Reality Interaction Techniques
		5.1 Introduction
		5.2 Tangible and Surface-Based Interaction
		5.3 Gesture-Based Interaction
		5.4 Pen-Based Interaction
		5.5 Gaze-Based Interaction
		5.6 Haptic Interaction
		5.7 Multimodal Interaction
		5.8 Multi-Display Interaction
		5.9 Interaction Using Keyboard and Mouse
		5.10 Virtual Agents
		5.11 Summary and Outlook
		References
	6 Interaction with AI-Controlled Characters in AR Worlds
		6.1 Populating AR Worlds with Virtual Creatures
			6.1.1 AI Characters in AR Literature
		6.2 Designing AI Characters for AI Worlds
			6.2.1 Categorization of AI Characters
				Trainer, Trainee, and Coaches
				Subject of an Examination
				Assistants and Companions
				Enemies and Opponent
			6.2.2 Architecture Base Components
			6.2.3 Appearance
				Human-Human Communication
				Human-AI Communication
				Conclusions for AR Characters
			6.2.4 Movement
			6.2.5 Reasoning
				Decision Trees
				Finite State Machines
				Goal-Oriented Behavior
				Utility AI
		6.3 Conclusion
		References
	7 Privacy and Security Issues and Solutions for Mixed Reality Applications
		7.1 The Mixed Reality Present
			7.1.1 Overview on Mixed Reality Processing
			7.1.2 Towards MR Mobility
		7.2 Security and Privacy Risks with Mixed Reality
			7.2.1 Risks with MR Data Processing
			7.2.2 Mobility and Privacy
		7.3 Protection Approaches for Mixed Reality
			7.3.1 Input Protection
			7.3.2 Output Protection
			7.3.3 Protecting User Interactions
			7.3.4 Device Protection
			7.3.5 Open Research Challenges
			7.3.6 Future Directions
		7.4 Towards Everyday MR Services
		References
Part III Hardware and Peripherals
	8 The Optics of Augmented Reality Displays
		8.1 Introduction to Augmented and Virtual Reality
		8.2 A Brief History of AR Displays
		8.3 The Basics of Visual Instrument Design
			8.3.1 The Human Visual System
			8.3.2 Optical Design Properties for AR Displays
		8.4 Optical Components of an AR Display
			8.4.1 Microdisplays as the Light Engine
			8.4.2 Radiometric Brightness Analysis for AR Displays to Guide Microdisplay Specifications
			8.4.3 A Brief Foray into Laser Scanning
			8.4.4 Imaging Optics and Combiners
		8.5 Optical Architectures and How They Work
		8.6 Areas for Improvement
		8.7 Components and Techniques for AR Displays of the Future
			8.7.1 Pupil-Steering and Eye-Tracking
			8.7.2 Freeform Optics
			8.7.3 Metasurfaces
		8.8 Conclusion
		References
	9 Tracking Systems for Augmented Reality
		9.1 Introduction
		9.2 Multisensor Integration
		9.3 Calibration Methods
			9.3.1 Notations
			9.3.2 Tip Tool Calibration
			9.3.3 Tracking the Same Target
			9.3.4 Hand–Eye Calibration
			9.3.5 Absolute Orientation
		9.4 Registration Methods
			9.4.1 Iterative Closest Point
			9.4.2 Point-Feature-Based Alignment
		9.5 Inertial Measurement Unit Calibration
			9.5.1 IMU Bias
			9.5.2 Sensor Fusion
			9.5.3 IMU–Camera Calibration
		9.6 Projector–Camera Calibration
			9.6.1 Pixel Mapping
			9.6.2 Spatial Calibration
		9.7 Optical See-Through Head-Mounted Display Calibration
			9.7.1 Interaction-Based Methods
			9.7.2 Interaction-Free Calibration
			9.7.3 Eye Tracking
		9.8 Evaluation Methods
			9.8.1 Objective Measurements
			9.8.2 Subjective Measurements
		9.9 Tracking Systems for Sensor Integration
			9.9.1 Mechanical Links
			9.9.2 Electromagnetic Sensors
			9.9.3 Inertial Measurement Units
			9.9.4 Flex Sensors
			9.9.5 Radio Signals
			9.9.6 Camera-Based Motion Capture Systems
				Passive Markers
				Active Markers
			9.9.7 Markerless Tracking
				Human Pose Tracking
				Facial Tracking
				Hand Tracking
				Thermal Tracking
			9.9.8 Projection-Based Sensing
				Spatial Division Code
				Spatial Scanning
		9.10 Applications
			9.10.1 Medical Applications
			9.10.2 Robotic Applications
			9.10.3 Entertainment Applications
		9.11 Conclusion
		References
	10 Embodied Interaction on Constrained Interfaces for Augmented Reality
		10.1 Resurgence of Wearable Computers
			10.1.1 Interaction with Today's Wearable AR Headsets
			10.1.2 Drawing a Parallel to Desktops and Smartphones
			10.1.3 The Constrained Interfaces on Wearable ARHeadsets
			10.1.4 Rethinking on the Constrained AR Interfaces
			10.1.5 Spotlights of the Chapter
			10.1.6 Structure
		10.2 Related Work
			10.2.1 Freehand Pointing on Constrained Hardware
			10.2.2 Text Entry on Constrained Screen Real Estate
			10.2.3 Optimized Text Entry Layout Design
			10.2.4 Summary
		10.3 TiPoint
			10.3.1 System Requirements
			10.3.2 Interaction Overview
			10.3.3 Interaction Approaches
				Freehand Mode
				Fast-Repetitive Mode
			10.3.4 Mid-air Interaction Strategy for Small-Screen Display
			10.3.5 Implementation
			10.3.6 System-Wide Implementation
			10.3.7 Application Scenarios
		10.4 HIBEY
			10.4.1 System Design
				Character Keys
				The Gesture of a Pointing Hand
			10.4.2 Uncertainty on Keyboard-Less Environment
				Probabilistic Method for Handling Imprecision
			10.4.3 Implementation and User Performance
			10.4.4 Application Scenarios
		10.5 TOFI
			10.5.1 System Implementation
			10.5.2 Optimization of the Keyboard Layout
				Maximizing the Goodness of Character Pair
				Maximizing the Familiarity with the QWERTY Layout
				Maximizing the Easiness of Force Keypad Interaction
				Maximizing the Comfort Level of the Finger Space
				Optimized Keyboard Layouts
			10.5.3 User Performance
			10.5.4 Application Scenarios
		10.6 Take-Home Message for AR Interaction Techniques
			10.6.1 Conclusions
			10.6.2 Future Outlook: Toward the Miniature and Subtle Interfaces
		References
	11 Networking and Cyber Foraging for Mobile Augmented Reality
		11.1 Mobile Augmented Reality Requirements and Cyber-Foraging
			11.1.1 Requirements of MAR Applications
			11.1.2 Network Capabilities and MAR
			11.1.3 Cyber-Foraging for AR
			11.1.4 Arising Challenges
			11.1.5 Performance Models
			11.1.6 Highlights of This Chapter
		11.2 Related Works
			11.2.1 Mobile Augmented Reality
			11.2.2 Generic Cyber-Foraging Systems
			11.2.3 Cyber-Foraging for MAR
			11.2.4 Network Protocols
				Audio and Video Protocols
				D2D Multimedia Protocols
				Improving General Performance
			11.2.5 Discussions
		11.3 Network Access
			11.3.1 Wireless Networks
				HSPA+ (High Speed Packet Access)
				LTE (Long-Term Evolution)
				LTE Direct
				Wi-Fi
				Wi-Fi Direct and Wi-Fi Ad Hoc
			11.3.2 Future Wireless Architectures and 5G: Promises and Challenges
			11.3.3 Upload to Download Ratio on Asymmetric Links: A Delicate Balance
				History and Future of Access Networks
				Are Symmetric Links Really Necessary?
		11.4 Infrastructure and Transport Protocol for MAR
			11.4.1 Classful Traffic
			11.4.2 Congestion Control, Fairness, and Graceful Degradation
			11.4.3 Enforcing Low Latency with Loss Recovery
			11.4.4 Multipath
			11.4.5 Multi-Server and Distributed Computations
			11.4.6 Security and Privacy
			11.4.7 Implementation Notes
		11.5 Improving Latency at the Application Layer
			11.5.1 Mobile AR Pipeline
			11.5.2 Commercial SDK Cloud Offloading Procedures
			11.5.3  Commercial SDK End-To-End Latency Analysis
			11.5.4 Discussion
		11.6 Conclusion
			11.6.1 Access Link
			11.6.2 Network and Transport Layer
			11.6.3 Application Layer
			11.6.4 Future Challenges
		References
Part IV Applications in Arts, Education, and Culture
	12 Augmented Reality in Arts Education
		12.1 Basic Concepts
			12.1.1 Augmented Reality and Arts Education: What Intersection Point?
			12.1.2 Arts Education Strategies in the Twenty-First Century
			12.1.3 Augmented Reality in Art Teaching at a European Level: What Developments?
		12.2 Augmented Reality for Meeting and Making Art
			12.2.1 Augmented Reality for Knowledge of Arts
				Augmented Reality-Based Art Learning in Museums
				Augmented Reality-Based Art Learning in Archaeological and Art Sites
			12.2.2 Augmented Reality-Based Learning and Creative Expression
		12.3 Digital Environments and Augmented Reality
			12.3.1 Augmented Reality as Third Space
			12.3.2 Augmented Reality for Creating Digital Artefacts
		12.4 The Language of Images and Augmented Reality
			12.4.1 The Audio-Visual Language
				The Language of the Texts
				The Language of Filming
				The Language of Lighting
				The Language of the Setting
				The Language of Characterization
			12.4.2 Body Language
				The Language of Editing
				The Language of Sound
				The Language of Music
				The Language of Graphics
			12.4.3 Educational Experiences
				Images and Learning
				Digital Images and Media Education
			12.4.4 Augmented Reality and Educational Experiences with Arts
			12.4.5 Augmented Reality: Art for the Sake of Art
		12.5 Digital Competences and Augmented Reality
			12.5.1 Augmented Reality and Digital Innovation
			12.5.2 Augmented Reality and Digital Competences of Museum Educators
		References
	13 Augmented Reality's Application in Education and Training
		13.1 Development of Augmented Reality
		13.2 Defining Augmented Reality
		13.3 Pedagogical Framing of Augmented Reality
			13.3.1 Location
			13.3.2 Task
			13.3.3 Role
		13.4 Limitations
			13.4.1 Technical Issues
			13.4.2 Location Detection
			13.4.3 Usability
			13.4.4 Pedagogy
			13.4.5 Health and Safety
		13.5 Future Directions for Educational Use of Augmented Reality
		References
	14 Augmented Reality in Sports and Physical Education
		14.1 Introduction
		14.2 Status Quo of PE
		14.3 Application of IT in PE
		14.4 Design of an AR-Assisted Learning System for PE
		14.5 Research Methods and Verification
		14.6 Learning Outcomes of AR-Assisted Learning
		14.7 Motor Skill Acquisition by Using AR-PEclass
		14.8 Learning Motivation Stimulated by AR-PEclass
		14.9 Conclusion
		References
	15 Potentiating Learning Through Augmented Reality and Serious Games
		15.1 Introduction
		15.2 Augmented Reality in Pedagogical Contexts
		15.3 Serious Games in Pedagogical Contexts
			15.3.1 Educational Learning Approaches
			15.3.2 Design of Serious Games for Education
			15.3.3 Types of Serious Games Models
			15.3.4 Serious Games and User Experiences
		15.4 Serious Games and Augmented Reality
		15.5 Case Studies
			15.5.1 Visualizing Platonic Solids in Augmented Reality
				Motivation
				Technology and Specifications
				Pilot Study
				Observations
				Discussion
				Remarks
			15.5.2 CodeCubes
				Motivation
				Technical Development
				Pilot Study
				Procedure
				Discussion and Results
				Study Limitations
				Remarks
			15.5.3 FootMath
				Motivation
				Technical Development
				Exploring FootMath
				Pilot Study
				Procedure
				Discussion and Results
				Remarks
		15.6 Closing Remarks
		15.7 Conclusions
		References
	16 Augmented Reality for Cultural Heritage
		16.1 Virtual Technologies for Cultural Heritage Legibility
		16.2 State of the Art on AR in CH Contexts
			16.2.1 Historical Background
			16.2.2 Usage of AR Between Real and Virtual
		16.3 Relevant Case Studies in CH: Different Approaches to AR
			16.3.1 A Selection of Case Studies
				Olympia, 2000
				Drawings on Glass at Carnuntum (Austria), 2011
				Jumieges 3D, Normandy (France), 2013
				ViaggiArte: Fontanelle Cemetery, Naples (Italy), 2019
				CEMEC: The Box of Stories, EU Project, 2015–2019
				The Santa Maria Antiqua Videomapping, Rome (Italy), 2015
				The Revealing Flashlight, EU Project, 2013
				iMARECULTURE, EU Project, 2016–2020
				ARETE, EU Project, 2019–2023
			16.3.2 Analytical-Comparative Summary of Use Cases
		16.4 Technical Limitations and Challenges of AR for CH
		16.5 Multisensorial AR Experiences
		16.6 Efficacy and Effectiveness of AR
			16.6.1 The Importance of User Experience Design for AR Systems
			16.6.2 Efficacy and Effectiveness of AR Systems in Relation to the Context of Use
		16.7 Directions and Future Perspectives
		References
	17 Augmented Reality in Holocaust Museums and Memorials
		17.1 Introduction
		17.2 Traditional Education of the Holocaust
			17.2.1 Ethical Considerations for Education of the Holocaust
		17.3 Multimedia Content for Holocaust Education
		17.4 Augmented Reality for Holocaust Education
		17.5 Ethics of Developing Augmented Reality Apps for Holocaust Museums and Memorials
		17.6 Visualization Methods for Digital Reconstruction
		17.7 Potential Applications of Augmented Reality for Holocaust Education
		17.8 Conclusion
		References
	18 Augmented Reality into Live Theatrical Performance
		18.1 Background
		18.2 Augmented Reality in Theatrical Performance
			18.2.1 ALICE Project
				Technical Components
				Technological Affordances in Performance
				Safety Considerations
				Sensor Considerations
				Data Considerations
				Considerations for Working with Performers
			18.2.2 What the Moon Saw
				Agentic Affordances of AR in Theater for Young Audiences
				Storytelling Affordances of AR
				Considerations for Public Interaction
				Multiple Forms of Public Engagement
				Actors, Audience, and AR
		18.3 Discussion
			18.3.1 Potentials for Nonlinear Storytelling
			18.3.2 Interfacing for Immersion
			18.3.3 Accessibility
			18.3.4 Future Directions
		18.4 Conclusion
		References
	19 How do Tourists Evaluate Augmented Reality Services? Segmentation, Awareness, Devices and Marketing Use Cases
		19.1 Introduction
		19.2 Holistic Augmented Reality Marketing in Tourism
			19.2.1 Augmented Reality Marketing
			19.2.2 Augmented Reality Marketing in Tourism: Use Cases
				AR in the Pre-Booking and Information-Gathering Stage
				On-Site AR Experiences
			19.2.3 Augmented Reality Marketing in Tourism: Prior Research
		19.3 The Role of Augmented Reality in Tourism: Research Questions
		19.4 Study
			19.4.1 Research Design and Methodology
		19.5 Results
		19.6 Discussion
			19.6.1 General Conclusion
				Be Aware that Consumers are Not Aware
				AR is Just Getting Started
				Focus on Individual Tourist Needs
		References
Part V Applications in Engineering and Science
	20 Augmented Reality Uses and Applications in Aerospace and Aviation
		20.1 History of Augmented Reality in Aerospace
		20.2 Applications in Navigation and Guidance
			20.2.1 Uses of AR in Manned Navigation and Guidance
			20.2.2 Unmanned Navigation and Guidance
			20.2.3 Air Traffic Management
		20.3 Applications in Engineering, Manufacturing, and Maintenance
			20.3.1 Engineering Design and Visualization
			20.3.2 Manufacturing and Maintenance Support
				Remote and Tele-maintenance
		20.4 Applications in Space Operations
		20.5 Future Application Concepts
			20.5.1 Airport Security
			20.5.2 Crew Task Support
			20.5.3 In-Flight Entertainment and Communication
			20.5.4 Augmented Reality and Artificial Intelligence
			20.5.5 Augmented Reality and Haptic Feedback
			20.5.6 Augmented Reality Applications for COVID-19 Support
		20.6 Concluding Remarks
		References
	21 Augmented Reality for Building Maintenance and Operation
		21.1 The Building Maintenance Instruction Manual
		21.2 Augmented Reality and Facility Management
		21.3 Systematic Literature Review
			21.3.1 Existing Studies
			21.3.2 Artifacts Categorization
		21.4 Methodology
			21.4.1 Identification of the Problem
			21.4.2 Design and Development
			21.4.3 Artifact Evaluation
			21.4.4 Explicitness of Learning
			21.4.5 Generalization to a Class of Problems
		21.5 Design and Development
			21.5.1 Artifact Design
			21.5.2 Artifact Development
			21.5.3 Artifact Publication
		21.6 Evaluation and Findings
			21.6.1 General Workload Measurement: NASA TLX
				Total Sample Characterization
				Workload and Factor Analysis
			21.6.2 Workload Analysis Considering Perception Filters
		21.7 Generalization
		21.8 Conclusion
		21.9 Data Availability
		References
	22 An Augmented Reality Platform for Interactive Finite Element Analysis
		22.1 Introduction
			22.1.1 Brief Overview of FEA
			22.1.2 Augmented Reality
			22.1.3 Research Motivations and Objectives
		22.2 Research Background
			22.2.1 Interactive FEA in VR
			22.2.2 Numerical Simulation and Scientific Visualization in AR
			22.2.3 Real-Time Finite Element Modeling
			22.2.4 Discussion
			22.2.5 Summary
		22.3 FEA-AR Integrated System
			22.3.1 System Design Considerations
			22.3.2 System Architecture
			22.3.3 AR Environment Setup
				Hardware Configuration
				Interaction Tools
				Coordinate Systems and Transformation
				Object Selection Techniques
			22.3.4 Summary
		22.4 Visualization and Exploration of FEA Results
			22.4.1 Scientific Visualization with VTK
			22.4.2 Scientific Visualization in AR1pt
				Integration of VTK and AR
				Occlusion Handling
			22.4.3 Data Manipulation and Exploration
			22.4.4 Summary
		22.5 Real-Time FEA in AR
			22.5.1 Real-Time FEA Solution
			22.5.2 Computation of Inverse Stiffness Matrix
				Matrix Inversion with PCG Method
				Matrix Inversion Using External FEA Program
			22.5.3 Load Acquisition
				Load Management and Conversion
				WSN Configuration
				Application of Virtual Loads
			22.5.4 System Workflow and Time Synchronization
			22.5.5 Summary
		22.6 Interactive Model Modification
			22.6.1 Adding Geometric Models
			22.6.2 Local Mesh Refinement
			22.6.3 Summary
		22.7 System Implementation and Case Studies
			22.7.1 Case Study of a Step Ladder
				Model Preparation
				Data Visualization and Exploration
				Real-Time Simulation
				Re-Analysis with Model Modification
				Local Mesh Refinement
				System Response Time
			22.7.2 Case Study of a Proving Ring
			22.7.3 A Prototype Application for Education
			22.7.4 Summary
		22.8 Conclusions and Future Readings
			22.8.1 Some Open Issues
				Enrich the Interactive Model Modification Methods
				Apply Model Reduction Techniques for Efficient Analysis
				Adapt the System to Mobile AR Platforms
				Real-Time Simulation of Soft Objects in AR
				Structural Health Monitoring with Finite Element Model Updating
		References
	23 Augmented Reality in Maintenance: A Review of the State-of-the-Art and Future Challenges
		23.1 Augmented Reality Applications in Maintenance
		23.2 Summary of Reviews of Augmented Reality in Maintenance
		23.3 Literature Review
			23.3.1 Applications
			23.3.2 Contextual Awareness
			23.3.3 Tracking
			23.3.4 Human Motion Tracking
			23.3.5 User Interface and Interaction
			23.3.6 Hardware
		23.4 Discussion
			23.4.1 Feasibility of AR Application in Industrial Maintenance
			23.4.2 Uniform Implementation Method for AR in Maintenance
		23.5 Research Gaps Based on Current Technological Developments
			23.5.1 A Unified and Robust Design Approach to Achieve Contextual Awareness
			23.5.2 UI Design for AR in Remote Maintenance
			23.5.3 IoT and Digital Twin Integration
			23.5.4 Improvements in User Tracking
			23.5.5 Ergonomic Assessment of Current Maintenance Practices
			23.5.6 Potential Future Research Focus Areas
				Closed-Loop AR-Assisted Maintenance Systems
				Context Awareness of AR-Based Maintenance Systems
				Automated Authoring of AR-Assisted Maintenance Systems
				Ergonomic Consideration and Tracking
				Development of More Robust Remote AR Maintenance Systems
				IoT and Industry 4.0 Integration
		23.6 Conclusion
		References
	24 Augmented Reality for Maintenance and Repair
		24.1 Introduction
		24.2 AR for Maintenance
		24.3 AR Solutions for Repairs and Troubleshooting
			24.3.1 AR and Industry 4.0
			24.3.2 Techniques and Methodologies Behind AR-Aided Servicing
			24.3.3 Augmentation Strategies for AR-Based Guidance to Repair Procedures
			24.3.4 Near-Future Scenarios: Collaborative Repair and Troubleshooting ThroughAR and Virtual Assistants
		24.4 Design of a Virtual Toolbox: TheiEngine Framework
			24.4.1 Layered Architecture of iEngine
				User Layer
				Hardware Layer
				Low-Level/Interface Layer
				Middle-Level/Manager Layer
				High-Level/Decision-Making Layer
			24.4.2 iEngine Module Architecture
				Modules
				Services
				Plugins
				Packages
			24.4.3 Novelty of the Developed Approach in iEngine
		24.5 Use Case and Experimentation
			24.5.1 VTE Modules
			24.5.2 VTE Application AI Module
				Finite State Machines
				Decision Trees
			24.5.3 VTE Final Result
		24.6 Conclusions
		References
	25 Augmented Reality for Naval Domains
		25.1 Historical Review of Selected Research
			25.1.1 Implementations of the Head-Up Display Concept
				Aircraft Pilot
				Ship Captain
				Dismounted Personnel
			25.1.2 Maintenance and Repair Applications
				Aircraft Manufacturing
				Military Maintenance
		25.2 Research and Development Efforts
			25.2.1 Dismounted Infantry Training
				Research at the Naval Research Laboratory
				Development via the Office of Naval Research
				Evaluation and Testing by the Marine Corps
				Discussion of Marine Training Application
			25.2.2 Other Applications to Military Operations
				GunnAR
				Augmented Ship Transits for Improved Decision-Making
				Virtual Scientist
				Battlespace Visualization
			25.2.3 Maintenance and Repair
				Service Maintenance Augmented Reality Tools
				In-Service Engineering Agent of the Future
				Ocean AR
				Local Maintenance
				Remote Maintenance Aid
			25.2.4 Other Applications and Technologies
				Vestibular Therapy Using AR
				Rapid Prototypes
				Human Factors
				Color Perception
		25.3 Discussion
			25.3.1 Augmented Reality in the Navy: A Roadmap
			25.3.2 Human Factors Engineering Technical Advisory Group
			25.3.3 Lessons Learned
		25.4 Conclusions and Future Directions
		References
	26 Augmented and Mixed Reality for Shipbuilding
		26.1 Introduction
		26.2 State of the Art
			26.2.1 Shipbuilding Welding
			26.2.2 Shipbuilding Painting
			26.2.3 Shared Information
			26.2.4 Step-by-Step Guidance
			26.2.5 Design and Construction Assistance
			26.2.6 Commercial Developments
		26.3 Potential Shipbuilding Tasks to Be Enhanced with AR/MR
			26.3.1 Quality Control
			26.3.2 Guided Manufacturing
			26.3.3 Product and Tool Tracking
			26.3.4 Warehouse Management
			26.3.5 Predictive Maintenance
			26.3.6 Augmented and Mixed Reality Communications
			26.3.7 Hidden Area Visualization
			26.3.8 Monitoring and Interaction with IIoT Devices
			26.3.9 Easy Interaction with Advanced Industrial Software
			26.3.10 Structure Visualization
			26.3.11 Training
			26.3.12 Product and Tool Maintenance
		26.4 AR/MR Architectures for Shipbuilding Applications
			26.4.1 Traditional AR/MR Communications Architectures
			26.4.2 Advanced Communications Architectures
		26.5 AR/MR Hardware and Software for Shipbuilding Applications
			26.5.1 Ideal AR/MR Device Characteristics
			26.5.2 AR/MR Devices for Shipbuilding Applications
			26.5.3 Software AR/MR Frameworks for Shipbuilding
			26.5.4 Lessons Learned from the Shipbuilding State of the Art
		26.6 Main Challenges for the Development of AR/MR Shipbuilding Applications
		26.7 Conclusions
		References
	27 Augmented Reality for Remote Assistance (ARRA)
		27.1 Introduction
		27.2 Background
		27.3 ARRA
			27.3.1 ARRA: A Practical Example
			27.3.2 Technical Development
		27.4 Test Design and Methodology
			27.4.1 Validation Case Study
			27.4.2 Quantitative Validation Test Methodology
		27.5 Analysis and Results
		27.6 Discussion
		27.7 Conclusion and Future Work
		References
Part VI Applications in Health Science
	28 Augmented Reality for Computer-Guided Interventions
		28.1 From Medical Image Computing to Computer-Assisted Interventions
		28.2 Imaging and Anatomical Modeling
		28.3 Surgical Vision
		28.4 Registration and Pose Estimation
		28.5 Physics-Based Modeling
			28.5.1 Soft Tissue Biomechanics
			28.5.2 Strategies for Real-Time Computation
			28.5.3 Beyond Surgery
		28.6 Visualization and Perception
		28.7 Related Topics
			28.7.1 Training and Knowledge Transfer
			28.7.2 Surgical Planning
		28.8 The Future of AR in Medicine
		References
	29 Youth and Augmented Reality
		29.1 Introduction
		29.2 Augmented Reality in Video Games
			29.2.1 Brief History of Augmented Reality Video Games (ARGs)
			29.2.2 Modern ARGs Utilize Smartphone Technology
			29.2.3 Benefits of Augmented Reality Video Games
			29.2.4 Downsides to Augmented Reality Games' Growing Popularity
			29.2.5 Safety Precautions When Using Augmented Reality Games
			29.2.6 The Future of Augmented Reality Games
		29.3 Augmented Reality and Social Media
			29.3.1 Snapchat
			29.3.2 Facebook, Messenger, and Instagram
			29.3.3 TikTok
			29.3.4 Up and Coming AR: Pinterest
			29.3.5 Precautions and Risks of AR Use on Social Media Among Children and Adolescents
			29.3.6 The Future of AR on Social Media
		29.4 Augmented Reality in Sports
			29.4.1 Enhancing Athletic Training
			29.4.2 Spectator Interactions
			29.4.3 Limitations of Augmented Reality in Sports
			29.4.4 The Future of Augmented Reality in Sports
		29.5 Augmented Reality in Art
			29.5.1 Art Appreciation and Experience
			29.5.2 Augmented Reality, Art, and Young Children
			29.5.3 Limitations of Art and AR
			29.5.4 Future of Art and AR
		29.6 Augmented Reality in Education
			29.6.1 Preschool
			29.6.2 Elementary School
			29.6.3 Middle School
			29.6.4 High School
			29.6.5 College
			29.6.6 Graduate Education and Vocational Training
				Medical School Students
				Physicians and Clinicians
				Patients
				Other Vocations
			29.6.7 Augmented Reality and the Disabled
			29.6.8 Cautions
		29.7 Augmented Reality in Medical Practice
			29.7.1 Educational Opportunities
			29.7.2 Therapeutic Uses
				Medical Procedures
				Patient Communication, Management, and Experience
				Nonprocedural Medical Uses
				Mental Health
				Health/Wellness Goals
		29.8 Augmented Reality Safety Precautions
			29.8.1 Safety Prior to AR Use
				Indoor Considerations
				Outdoor Considerations
				Driving
				Distracted Driver Prevention AR Applications
			29.8.2 Safety Measures During AR Use
				Injuries and Risks from Spatial Ware and Body Position
				Eye Strain
				Deep Vein Thrombosis
				Injuries from Repetitive Movements
			29.8.3 Mental Health
		29.9 Considerations and Causes for Concern
			29.9.1 Age Considerations for Augmented Reality
				Piaget Theory Age-Related Concerns
				AAP Recommendations and AR-Specific Supplementation by Authors
			29.9.2 Gender Considerations
			29.9.3 Pandemic Considerations
			29.9.4 Accessibility for All
		29.10 Conclusion
		References
	30 Augmented Reality-Assisted Healthcare Exercising Systems
		30.1 Augmented Reality-Assisted Healthcare Exercising Systems
		30.2 Related Works
			30.2.1 Upper-Extremity Healthcare Exercising Systems
				Conventional Methods
				VR-Based Systems
				Haptic-Based Systems
				AR-Based Systems
			30.2.2 Upper-Extremity Movement Tracking Methodologies
				Sensor-Based Tracking
				Bare-Hand Tracking
			30.2.3 Assessment of User's Upper-Extremity Functions Recover
				Outcome Measurements
				Glove-Based Devices for Motor Functions Assessment
			30.2.4 Discussion
		30.3 System Overview
		30.4 Monitoring Module
			30.4.1 Motion Tracking
			30.4.2 AR-Based Data Glove
			30.4.3 Vibration Wrist Band
		30.5 Exercise Module and Scoring Module
			30.5.1 Flaccid Stage
				ROM Exercise for Finger
				ROM Exercise for Shoulder
			30.5.2 Synergy Stage
				Hole Peg Test
				Reach-to-Grasp Exercise
				Scoring Module for Synergy Stage
			30.5.3 Activities of Daily Living Stage
				Opening Doors
				Arranging the Bookcase
				Drawing Water from Faucets
				Scoring Module for the Daily Activities
		30.6 Usability Experiment
			30.6.1 Method
			30.6.2 Participants
			30.6.3 Experiment Procedure
			30.6.4 Design of Pre-test and Post-test
			30.6.5 Results and Discussion
		30.7 Conclusion and Future Work
		References
	31 Augmented Reality for Cognitive Impairments
		31.1 Introduction
		31.2 Background Research
			31.2.1 Devices and Tools for AR
			31.2.2 Computing Paradigms for AR
		31.3 Current Development
			31.3.1 Human Memory System
			31.3.2 AR Applications in Research of Human Memory and Cognition
				Historical Background
				AR-Enhanced Memory Aid Systems
			31.3.3 AR Applications in Cognitive Load Research
				Cognitive Load
				AR in Cognitive Load Research
			31.3.4 AR Applications for Restoring Perception
				Perception-Related Cognitive Impairments
				AR-Enhanced Applications in Research of Perception Impairments
			31.3.5 Issues and Challenges in the Application of AR for Cognitive Impairments
		31.4 Conclusions
		References
Part VII Convergence with Emerging Technologies
	32 The Augmented Reality Internet of Things: Opportunities of Embodied Interactions in Transreality
		32.1 Introduction
		32.2 Background
			32.2.1 Augmented, Virtual, and Mixed Reality
			32.2.2 Artificial Intelligence and Ubiquitous Computing
				Intelligent Virtual Agents
				Internet of Things
				Smart Connected Environments
		32.3 The Augmented Reality Internet of Things in Transreality
			32.3.1 Convergence of Augmented Reality with Artificial Intelligence and Ubiquitous Computing
				Artificial Intelligence and Ubiquitous Computing
				Augmented Reality and Artificial Intelligence
				Augmented Reality and Ubiquitous Computing
			32.3.2 Transreality and The Augmented Reality Internet of Things
		32.4 AR-IoT Framework and Interaction Design
			32.4.1 Object-Centric AR-IoT Data Management
			32.4.2 Scalable AR-IoT Recognition and Tracking
			32.4.3 Context-Based AR-IoT Interactions and Content Interoperability
			32.4.4 AR-IoT Framework Evaluation
		32.5 Opportunities of Embodied Interactions in AR-IoT Environments
			32.5.1 Embodied Interactions in AR-IoT Environments
			32.5.2 Embodied AR-IoT Agent Prototypes
			32.5.3 Embodied AR Agent Insights
			32.5.4 Potential Use Cases
		32.6 Conclusions
		References
	33 Convergence of IoT and Augmented Reality
		33.1 Introduction
			33.1.1 Mixed Reality: A Parallel to the IoT Ecosystem
			33.1.2 Toward XRI: A Growing Trend in Hybrid Mixed Reality IoT Systems
			33.1.3 Chapter Overview
		33.2 A Multidisciplinary Taxonomy for XRI Systems
			33.2.1 XRI Taxonomy: Thematic Literature Review
			33.2.2 Relating the XRI Taxonomy to XRI System Design Needs from a Multi-disciplinary Perspective
		33.3 Contextual Reality (CoRe) Frameworks for XRI System Design
			33.3.1 Designing Proof-of-Concept XRI Systems with CoRe
			33.3.2 CoRe Framework Insights in Relation to the XRI Taxonomy Perspectives
		33.4 Discussion
			33.4.1 Directions for Future Work
		33.5 Conclusion
		References
	34 Digital Twin and Extended Reality: Strategic Approach and Practical Implementation
		34.1 Introduction
		34.2 DTs Enabling Technologies
			34.2.1 Enabling Technologies of DT
				Cyber-Physical Systems
				Internet of Things
				Simulation
				Extended Reality
				Augmented Reality
			34.2.2 Digital Twins
		34.3 Research Methodology
			34.3.1 Literature Review
				Planning the Review
				Conducting the Review
				Reporting the Review
		34.4 Results
			34.4.1 Findings from the Literature Review
				DTs for Asset Management in Manufacturing
				DTs for Process Improvement
				DTs for Factories
				DTs for People
				DTs for Product
				DTs Benefits
				AR in DT Applications
				Lifecycle Perspective
			34.4.2 DT Strategic Tool
				Structure of the Tool
				Technological Level
		34.5 Practical Implementation of a Combined DT-ER Solution
		34.6 Discussion
		34.7 Conclusion
		References
	35 Digital Twins as Foundation for Augmented Reality Applications in Aerospace
		35.1 AR-Based System Modeling in Aerospace
		35.2 Background
			35.2.1 Digital Twin
			35.2.2 Digital Twins in Aerospace
			35.2.3 Model-Based System Engineering
		35.3 AR Digital Twins in Aerospace Applications
		35.4 Challenges in AR Digital Twin Implementations
			35.4.1 Overlay Precision
			35.4.2 Interaction
		35.5 Dedicated AR Applications
			35.5.1 From Diagrams to AR in Spacecraft Design
			35.5.2 On-Orbit Servicing
				ar-Based Guidance and Training
			35.5.3 Remote Collaboration and Maintenance
		35.6 Closing Remarks
		References
Index