Cybernetic Avatar: 2025th Edition

Preface
Contents
About the Editors
1 Introduction: Cybernetic Avatar
	1.1 Avatar as a New Medium
	1.2 The Vision of Moonshot Goal 1 Avatar Symbiotic Society Project
	1.3 How Will CAs Change Society?
	1.4 Challenges of CAs Research and Development
	References
2 Development of Cybernetic Avatars with Humanlike Presence and Lifelikeness
	2.1 Introduction
	2.2 Research and Development of Cybernetic Avatar with Humanlike Presence
		2.2.1 Research and Development of Human-Engaged Robots and Avatars
		2.2.2 Operating Interface for HP-CA
		2.2.3 Research and Development of HP-CA in Our Moonshot Project
	2.3 Research on the Cognitive Aspects of a High Presence Teleoperation Interface
		2.3.1 Research Objective
		2.3.2 Proposed System Concepts
		2.3.3 Experiment
		2.3.4 Summary
	2.4 Research and Development of a Mobile Humanoid CA
		2.4.1 Realistic Head Unit for the Humanoid CA: Development and Features
		2.4.2 Operation Interface for the Humanoid CA Operation
		2.4.3 Mobility Mechanisms in Humanoid CA
		2.4.4 Potential Applications of Humanoid CA
		2.4.5 Concluding Remarks
	2.5 Development of a Huggable CA
		2.5.1 Background of a Huggable CA
		2.5.2 Moffuly-MS: A Huggable CA with a Large Stuffed-Toy Appearance
		2.5.3 Avatar Hiro-Chan: A Huggable CA with a Baby-Like Stuffed-Toy Appearance
		2.5.4 Potential Use Cases
		2.5.5 Summary
	2.6 Development of Life-Like CA and Mechanisms for Collaborative Conversation
		2.6.1 Collaborative Dialogue System of Multiple CAs for Simultaneous Dialogue Services at Multi-locations
		2.6.2 Extending Multiple CAs Dialogue System to Online Conversation
		2.6.3 Summary
	2.7 A Study on Generating Natural CA Motion Without Being Aware of Teleoperation
		2.7.1 Methods to Generate Humanlike CA Motions
		2.7.2 Modeling Motions Expressing the Operator’s Character
	2.8 Development of an Interaction Behavior Learning Method for CAs
		2.8.1 Modeling Dyadic Interaction
		2.8.2 Generative Model for Dyadic Interaction
		2.8.3 Feature Space of the Interaction
		2.8.4 Semi-autonomous Teleoperation System for Cybernetic Avatars
		2.8.5 Summary
	References
3 Spoken Dialogue Technology for Semi-Autonomous Cybernetic Avatars
	3.1 Introduction
	3.2 System Concept and Overview
		3.2.1 Architecture and Process Flow
		3.2.2 Advantages and Goals
		3.2.3 Technical Challenges
	3.3 Acoustic Processing for Real-World Human–Avatar Interaction
		3.3.1 Acoustic Environment for CA’s Ear
		3.3.2 Audio Source Separation: Basic Theory
		3.3.3 Audio Source Separation for Human–Avatar Interaction
		3.3.4 Future Directions
	3.4 Speech Processing for Real-Time Human–Avatar Interaction
		3.4.1 Automatic Speech Recognition
		3.4.2 Speech Synthesis
		3.4.3 Turn-Taking
		3.4.4 Backchannel Generation
		3.4.5 Filler Generation
		3.4.6 Shared Laughter Generation
		3.4.7 Attentive Listening System
	3.5 Dialogue Processing for CAs
		3.5.1 Handover in Spoken Dialogue
		3.5.2 Spoken Dialogue Technology for the Handover of Dialogue
		3.5.3 Future Directions
	3.6 Infrastructure of CG-CA
		3.6.1 CG-Based Cybernetic Avatar
		3.6.2 Design Factors of CG Agents
		3.6.3 Design Factors of CG Avatars
		3.6.4 Development of a CG-CA System for Semi-autonomous CAs
		3.6.5 Architecture of the CG-CA System
	3.7 System Implementations
	References
4 Human-Level Knowledge and Concept Acquisition
	4.1 Introduction
	4.2 Spatio-temporal Understanding of the Environment
		4.2.1 Background
		4.2.2 Point Cloud Matching for the Reconstruction of Non-rigid Objects
		4.2.3 Point Cloud Registration for Reconstruction of Non-rigid Scenes
		4.2.4 Summary
	4.3 Continuous Learning and Memory Mechanism
		4.3.1 Artificial Representation for Continuous Learning
		4.3.2 Artificial Constant Representation for Visual Signals
		4.3.3 Encoding and Decoding Knowledge from Language
	4.4 Research on Time Series Analysis
		4.4.1 Loss Function
		4.4.2 Model Structure
		4.4.3 Future Directions
	4.5 Language Learning and Generation and Its Interpretation
		4.5.1 Cross-Lingual Communication Support
		4.5.2 Improving the Usability of Large Language Models (LLMs)
		4.5.3 Summary
	4.6 Cybernetic Avatars for Medical Applications
		4.6.1 Background
		4.6.2 Weakly Supervised Learning Method for Chest X-Ray Images
		4.6.3 Domain Adaptive Multiple Instance Learning for Pathological Images
		4.6.4 Mental Health
	References
5 Cooperative Control of Multiple CAs
	5.1 Introduction
	5.2 Development of Flexible CA Control Technologies
		5.2.1 Multiple-CA Control Using LLMs
		5.2.2 Shared CA Control Based on User-Intention Estimation
	5.3 Hierarchical Control for Autonomous CAs
		5.3.1 Acquisition of Spatiotemporal Structure of Environment by CAs
		5.3.2 Emergent Communications Based on Gaussian Process Latent Variable Model (GPLVM)
		5.3.3 Cooperative Control Based on Emergent Communication for Autonomous CAs
	5.4 Research and Development of Embodied Dialogue CAs
		5.4.1 Multimodal Language Understanding in “Retrieve” Tasks
		5.4.2 Multimodal Language Understanding in “Carry” Tasks
	5.5 Research and Development of Daily-Physical-Support CAs
		5.5.1 Integration of Common Sense and Local knowledge
		5.5.2 Active Semantic Mapping and Exploration in Local Environment
		5.5.3 Understanding Situated Language
		5.5.4 Summary
	5.6 Tactile Sensing and Control for CA Manipulation
		5.6.1 Overview of CA Manipulation
		5.6.2 Proximity Sensor for CA’s Hand
		5.6.3 Operational Assistance Based on Proximity Sensing
		5.6.4 Integration with Teleoperation System
	5.7 Harmonized Control of Autonomous CAs Based on Voluntary BMIs
		5.7.1 Brain–Machine Interfaces
		5.7.2 Significance of Harmonized Control Between CAs and BMIs
		5.7.3 Implantable Brain–Machine Interfaces and Their Target Diseases
		5.7.4 Core Technology of Implantable BMI
		5.7.5 Harmonized Control of Autonomous CA and Voluntary BMI
		5.7.6 Extending Diversity of Devices Connected to BMI
	References
6 Development of the CA Platform
	6.1 Introduction
	6.2 What is a Cybernetic Avatar Platform (CAPF)?
		6.2.1 Cybernetic Avatar Platform
		6.2.2 Implementation of Cybernetic Avatar Platform
		6.2.3 Discussion
	6.3 User/CA Activity Monitoring and Management
		6.3.1 What is Activity Monitoring and Management?
		6.3.2 Roles of Activity Monitoring and Management
		6.3.3 Mechanism that Supports Activity Monitoring
		6.3.4 Application of Activity Monitoring
		6.3.5 Discussion
	6.4 Standardization for Service Robots and Cybernetic Avatars
		6.4.1 Standardization of Robot Technology at OMG
		6.4.2 Standardization for Cybernetic Avatars
		6.4.3 Standardization for Cybernetic Avatars and Their Social Implementation
	References
7 Multidisciplinary Investigation on How Avatars and Devices Affect Human Physiology
	7.1 Introduction
	7.2 Integrative Bioanalytics on Human–Avatar Interactions
		7.2.1 Homeostasis in Biology
		7.2.2 Introduction into Multi-omics Analyses
		7.2.3 Transcriptomics
		7.2.4 Confounding Factors
		7.2.5 The Need for Omics Analyses in the Field of Cybernetic Avatars
		7.2.6 Learning from Gaming Issues
	7.3 A Survey of Biomarkers to Assess the Impact of Cybernetic Avatars and Devices on Users
		7.3.1 Metabolomics
		7.3.2 New Technologies for Metabolomic Analysis
		7.3.3 Examples of Biomarker Discovery Studies
		7.3.4 Metabolomics to Assess the Effects of Cybernetic Avatar Use
	7.4 Brain Response Analysis During the Use of Avatars and Devices
		7.4.1 The Proteus Effect
		7.4.2 Brain in Virtual Environments
		7.4.3 Social Interaction in VR
	7.5 Designing Experimental Systems for the Investigation of Psychobiological Responses to Cybernetic Avatars and Devices
		7.5.1 Influence of Using New Technologies on Human
		7.5.2 Mental Workload During Operating an Avatar
		7.5.3 VR Simulator for Investigating the Psychological and Physiological Impact of an Avatar on Its Operator
		7.5.4 Investigation of Psychological and Physiological Influence During a Social Task
		7.5.5 Summary
	7.6 How to Create Health Indicators for Avatar Applications
		7.6.1 What Is Health: Definition
		7.6.2 Evaluation of Health
		7.6.3 Issues to Consider When Assessing the Health of Avatar Operators
		7.6.4 How to Confirm the Positive Effects of Using Avatars
		7.6.5 Summary
	References
8 Field Experiments in the Real World
	8.1 Introduction
	8.2 Field Experiment Platform for Cybernetic Avatars
		8.2.1 Requirements for Social-Field Experiments of CA
		8.2.2 CA’s Social-Field Experiment Platform
		8.2.3 Experiments that Utilize CA’s Social-Field Experiment Platform
		8.2.4 Collaborative Efforts for Social Implementation Through Corporate Partnerships
		8.2.5 Discussion
	8.3 5G Communication Systems for Cybernetic Avatars
		8.3.1 Motivation and Scope
		8.3.2 Field Experiments on Private 5G Communication Systems
	8.4 CA Systems in Field of Nursing Care
		8.4.1 Background
		8.4.2 Design
		8.4.3 Real-World Proof-of-Concept Testing
		8.4.4 Conclusion and Outlook
	8.5 Social-Field Experiments for Individuals with Developmental Disorders and Depressive Disorders
		8.5.1 Our System: Supporters Operate a Robot
		8.5.2 Our System: Patients Operate a Robot
		8.5.3 Importance of Considering Individuality
		8.5.4 Intervention Using Online Training
		8.5.5 Applications to Other Psychiatric New Diseases
		8.5.6 Summary
	8.6 Social-Field Experiments for Older Adults
		8.6.1 Equipment
		8.6.2 Methods
		8.6.3 Results
		8.6.4 Discussion and Future Issues
	References
Chapter 9 Cybernetic Avatars and Society
	Abstract
	9.1 Introduction
	9.2 Implementing Cybernetic Avatars in a Society
		9.2.1 Legal, Ethical, and Socioeconomic Principles for CA
		9.2.2 Avatar Social Implementation Guidelines
		9.2.3 Use Case
		9.2.4 Steps Toward Disseminating CA Guidelines in Society
	9.3 Technologies for Implementing Socially Well-Accepted CAs
		9.3.1 Moral Interaction with Cybernetic Avatars
			9.3.1.1 Moral Interaction for Cybernetic Avatars
			9.3.1.2 Examples of Moral Interaction Research with Cybernetic Avatar
			9.3.1.3 Discussion
		9.3.2 Socio-cultural Aspects in Designing Avatar Behaviors
			9.3.2.1 Motivation
			9.3.2.2 Culturally Adaptive Agents and Robots
			9.3.2.3 Toward Designing Cultural and Ethical CAs
		9.3.3 Cooperative Social Perception–Intention–Action (PIA) Model for Cybernetics Avatars
			9.3.3.1 The Social PIA Model
			9.3.3.2 The Social PIA Model Extended to Cybernetics Avatars
			9.3.3.3 The Use Case of Social PIA in a Handover Collaborative Task
	9.4 Ethical Issues Concerning Cybernetic Avatars
		9.4.1 Work and Life in Avatar Symbiotic Society
		9.4.2 Gendering and Its Ethical Implications
		9.4.3 Summary
	9.5 Legal Issues Concerning Cybernetic Avatars
		9.5.1 What is an Avatar?
		9.5.2 What is a Cybernetic Avatar?
		9.5.3 Usage Aspects of CAs
		9.5.4 Legal Status of CA
		9.5.5 Corporate CAs
		9.5.6 Proposals for New ELSI Studies
			9.5.6.1 Establishing a Research System to Ensure and Maintain the Social Acceptability of CA in Avatar Life
			9.5.6.2 ELSI Issues
		9.5.7 Economic Security and CA R&D
			9.5.7.1 Policy Development for CA R&D (Policy Advocacy, IP Protection, International Strategy, and Standardization)
			9.5.7.2 Leading International Rulemaking
		9.5.8 CA R&D and the Legal System
	9.6 Use of CA for Election Campaign
		9.6.1 Can CA Be Used for Campaigning?
		9.6.2 New Technologies and Election Campaigns
			9.6.2.1 Use of the Internet in Election Campaigns
			9.6.2.2 Use of the Internet in Japanese Election Campaigns
			9.6.2.3 Amendments to the Public Offices Election Law
		9.6.3 CA and Campaigning
		9.6.4 Use of On-Screen Avatars
		9.6.5 Possibility of Using CA in Election Campaigns
	References