The Multimodal Learning Analytics Handbook

Contents
Part I Introduction to MMLA
	Introduction to Multimodal Learning Analytics
		1 Introduction
		2 Background and Brief History: What Have We Learned and Where Do We Go Now?
			2.1 MMLA Brief History and Community Development
			2.2 MMLA Growth and Activities
			2.3 MMLA Challenges
				2.3.1 Coalescing of Multimodal Data and Advanced Analysis Methods to Support Learning
				2.3.2 Methodological and Practical Breakthroughs Through “Multimodality” to Support Learning
				2.3.3 MMLA Capabilities to Support Teaching and Learning
				2.3.4 From Experimentation, to Use and Adoption of MMLA
		3 Contributions and Themes on MMLA
			3.1 MMLA for Design
			3.2 MMLA for Feedback and Regulation
			3.3 MMLA to Support Theory and Pedagogy
			3.4 MMLA Approaches, Architectures and Methodologies
			3.5 MMLA and Affective States
			3.6 Privacy and Ethics of MMLA
			3.7 The Past, Present, and Future of MMLA
		4 Conclusions and the Way Ahead
		References
			In This Book
Part II MMLA for Design
	Multimodal Learning Analytics and the Design of Learning Spaces
		1 Introduction
		2 Research on Learning Spaces: From Traditional to Computer-Based Analysis
		3 Examples of MMLA Studies Focused on the Effects of Learning Spaces
			3.1 The PELARS Project: Standing for Better Collaboration
			3.2 The SmartLET Project: Studying the Interplay of Table Design and Educational Level
			3.3 Moodoo: Characterising Teachers' Positioning in the Classroom
		4 Discussion
			4.1 MMLA Potential for Informing the Design of Learning Spaces in Future Research
			4.2 MMLA Limitations to Inform the Design of Learning Spaces
			4.3 Broadening Use of MMLA Systems
		5 Concluding Remarks
		References
Part III MMLA for Feedback and Regulation
	Multimodal Systems for Automated Oral Presentation Feedback: A Comparative Analysis
		1 Introduction
		2 Anatomy of a Multimodal System for Oral Presentation Automated Feedback
		3 Comparative Technical Analysis of OPAF Systems
			3.1 Existing Systems
			3.2 Recording
			3.3 Extraction
			3.4 Analysis
			3.5 Feedback
			3.6 Conclusions
		4 Evaluation and Deployment
			4.1 Evaluation
			4.2 Levels of Deployment
		5 Conclusions and Next Steps
		References
	Modeling the Complex Interplay Between Monitoring Events for Regulated Learning with Psychological Networks
		1 Introduction
		2 Studying Monitoring as a Part of Self-Regulated Learning
		3 Physiological Activity in Relation to Monitoring
		4 Implementing Network Analysis to Capture the Complex Process of Monitoring and Physiological Arousal
			4.1 Recent Research Utilizing Psychological Networks
			4.2 Illustrating Network Analysis Methods in Practice with Three Data Examples
		5 Discussion and Future Prospects
			5.1 Prospects of Multimodal Learning Analytics
			5.2 Prospects of Psychological Networks
		References
	The Role of Metacognition and Self-regulation on Clinical Reasoning: Leveraging Multimodal Learning Analytics to Transform Medical Education
		1 Introduction
			1.1 Assessing and Evaluating Competency in Clinical Reasoning
			1.2 Multimodal Learning Analytics in Medical Education
		2 Socio-Cognitive Cyclic Model of Self-regulated Learning
			2.1 Forethought Phase
			2.2 Performance Phase
			2.3 Self-reflection Phase
		3 Implications of Multimodal Learning Analytics to Improve Medical Education
			3.1 Detecting Metacognition and Self-regulated Learning Processes
			3.2 Modeling Metacognition and Self-regulated Learning Processes
			3.3 Tracing Metacognition and Self-regulated Learning Processes
			3.4 Fostering Metacognition and Self-regulated Learning Processes
		References
Part IV MMLA to Support Theory and Pedagogy
	Intermodality in Multimodal Learning Analytics for Cognitive Theory Development: A Case from Embodied Design for Mathematics Learning
		1 Introduction
			1.1 Overview of the MIT-P Project
			1.2 Theoretical Framework: Intermodal Perception
		2 Multimodal MIT-P Analyses: A Brief History
			2.1 Hand Movements
			2.2 Eye Movements
			2.3 RQA Analysis
		3 From Multimodal Gaze and Hand Movement to the Intermodal Emergence and Stabilization of Attentional Anchors: An RQA Case Study
			3.1 Research Question
			3.2 Methods
			3.3 Results
				3.3.1 RQA Analysis
		4 Discussion
			4.1 Interpretation of Findings
			4.2 Theoretical Implications
			4.3 Methodological Implications
			4.4 Practical Implications
			4.5 Limitations
			4.6 Future Directions
		5 Conclusion
		References
	Bridging the Gap Between Informal Learning Pedagogy and Multimodal Learning Analytics
		1 Background
		2 Pedagogy of Informal Learning
			2.1 Context
			2.2 Learning Theories and Pedagogical Approaches
				2.2.1 Behaviorism
				2.2.2 Cognitivism
				2.2.3 Constructivism
			2.3 Reflections
		3 Multimodal Learning Analytics
			3.1 Context
			3.2 Informal Learning Modalities
			3.3 Underpinning Technicalities
			3.4 Multimodal Learning Analytic Challenges
		4 Discussion
		5 Conclusion
		References
Part V MMLA Approaches, Architectures and Methodologies
	Multimodal Learning Experience for Deliberate Practice
		1 Introduction
		2 Multimodal Learning Theories
			2.1 Embodied Learning
			2.2 Deliberate Practice
		3 Engineering Aspect
			3.1 Architecture Overview
			3.2 Interaction Layer
			3.3 Data Layer
			3.4 Feedback Layer
			3.5 Task Layer
			3.6 The MLX System into Teaching
		4 Research Methodologies
			4.1 First Iteration
			4.2 Second Iteration
			4.3 Following Iterations
		5 Application Use Cases
			5.1 Presentation Trainer
			5.2 CPR Tutor
			5.3 Calligraphy Tutor
			5.4 Table Tennis Tutor
			5.5 Astronaut Training
			5.6 Commonalities and Differences
		6 Conclusions and Future Work
		References
	CDM4MMLA: Contextualized Data Model for MultiModal Learning Analytics
		1 Introduction
		2 Developing MMLA Solutions in Authentic Settings: Lifecycle and Challenges
		3 Review of Related Contextualized Data Models
		4 Contextualized Data Model for MultiModal Learning Analytics (CDM4MMLA)
			4.1 Information Model of CDM4MMLA
			4.2 Description of the CDM4MMLA
		5 Applying CDM4MMLA to Authentic MMLA Scenarios
			5.1 Case 1: MUlti-Modal Teaching and Learning Analytics (MUTLA) Dataset Scenario
			5.2 Case 2: The MULTISIMO Corpus Scenario
			5.3 Case 3: MMLA for a Secondary School English Course Scenario
		6 Discussion
		7 Conclusions and Future Work
		References
	A Physiology-Aware Learning Analytics Framework
		1 Introduction
		2 State of the Art
		3 A Framework for Physiology-Aware Learning Analytics
			3.1 A Stress-Sensitive Pedagogical Agent in PHYLA
			3.2 PlugIn for HRV Data
			3.3 Physiological Backend: Physiological Parameters of Stress
			3.4 A Dialogue Concept for a Stress-Sensitive Pedagogical Agent
		4 Evaluation
			4.1 Design
			4.2 Results
				4.2.1 Study Participants
				4.2.2 Hypothesis 1
				4.2.3 Hypothesis 2
				4.2.4 Hypothesis 3
		5 Conclusion, Discussion, and Future Work
		Appendix
		References
Part VI MMLA and Affective States
	Once More with Feeling: Emotions in Multimodal Learning Analytics
		1 Introduction
		2 What Are Emotions?
		3 Emotions and Learning
			3.1 Emotions and Learning Outcomes
				3.1.1 Emotions and Learning Processes
		4 Measuring Emotions
		5 An Applied Example
			5.1 Data Sources
			5.2 Analyses and Workflow
				5.2.1 Statistical Analyses
			5.3 Results
		6 Discussion
			6.1 Valence Based Results
			6.2 Discrete Emotions-Based Results
			6.3 Challenges in Multi-Modal Affect Detection
		7 Conclusion
		References
Part VII Privacy and Ethics of MMLA
	The Evidence of Impact and Ethical Considerations of Multimodal Learning Analytics: A Systematic Literature Review
		1 Introduction
		2 Background and Related Work
			2.1 Ethical Considerations of Learning Analytics
		3 Methodology
		4 Results
			4.1 Data Modalities Used in MMLA Research
			4.2 Existing Evidence on the Use of MMLA to Support Educational Outcomes
			4.3 Ethical Considerations Highlighted and Addressed in MMLA Research
		5 Discussion
			5.1 Limitations
		6 Conclusions
		Appendix A
			WOS
			Scopus
			ACM
			IEEE
		Appendix B
		Appendix C
		Appendix D
		Appendix E
		References
Part VIII The Past, Present, and Future of MMLA
	Sensor-Based Analytics in Education: Lessons Learned from Research in Multimodal Learning Analytics
		1 Introduction
		2 Sensor-Based Analytics in Education
			2.1 SBA Qualities
			2.2 SBA Objectives
			2.3 SBA Challenges (Barriers of Adoption)
			2.4 SBA Opportunities
		3 The Role of SBA in Different MMLA Case Studies
			3.1 Case Study 1: SBA to Improve Accuracy of Learner Models
			3.2 Case Study 2: SBA to Capture Information Unobtrusively
			3.3 SBA and Embodied Learning
		4 Discussion: Towards SBA Adoption in Education
			4.1 Future Research on Methodological and Practical Aspects
			4.2 Future Research on Ethical Aspects
		Appendix: Categorization of Challenges and Opportunities, Across the Three Case Studies
		References
	Framing the Future of Multimodal Learning Analytics
		1 Introduction
		2 MMLA Research Paradigms
			2.1 MMLA for Theorizing
			2.2 MMLA for Practice
			2.3 MMLA for Interactivity
			2.4 Paradigms Summary
		3 Realizing Ethical Practices Across Different Aspects of an MMLA Research Project
			3.1 Data Collection: Multimodal Data Control/Data Ownership
			3.2 Data Analysis: Limitations in Prediction from Multimodal Data/Commitment to Fair and Ethical Language When Talking About Research Participants
			3.3 Data Dissemination: Transparency and Benefit/Moving Away from Research as an Extractive Process
			3.4 Commitments Summary
		4 Re-conceptualizing Learning Through an MMLA Perspective
			4.1 Methods for Data Analysis with Increased Data Privacy and Control
			4.2 Developing New Standards for Non-traditional Metrics
			4.3 Thinking About These Standards over Different Time Scales, Levels of Granularity, and Contexts
			4.4 Moving Beyond Randomized Control Trials as the Gold Standard
			4.5 Embracing Deep, Nuanced, and Potentially Divergent Pictures of the Learner
		5 Conclusion
		References
Index