Advances in Natural Multimodal Dialogue Systems (Text, Speech and Language Technology)

Contents......Page 6
Preface......Page 11
1. Introduction......Page 13
2. Chapter Presentations......Page 14
3. NMIE Contributions by the Included Chapters......Page 19
4. Multimodality and Natural Interactivity......Page 28
References......Page 31
Part I: Making Dialogues More Natural: Empirical Work and Applied Theory......Page 32
1. Introduction......Page 33
2. Embodied Conversational Agents......Page 35
3. Social Dialogue......Page 39
4. Related Work......Page 42
6. A Study Comparing ECA Social Dialogue with Audio-Only Social Dialogue......Page 50
7. Conclusion......Page 58
References......Page 59
1. Introduction......Page 65
2. Hosting Activities......Page 66
3. What is Engagement?......Page 67
4. First Experiment in Hosting: A Pointing Robot......Page 69
5. Making Progress on Hosting Behaviours......Page 72
6. Engagement for Human-Human Interaction......Page 73
7. Computational Modelling of Human-Human Hosting and Engagement......Page 80
8. A Next Generation Mel......Page 83
References......Page 84
Part II: Annotation and Analysis of Multimodal Data: Speech and Gesture......Page 87
1. Introduction......Page 88
2. Structure of FORM......Page 89
3. Annotation Graphs......Page 94
4. Annotation Example......Page 95
5. Preliminary Inter-Annotator Agreement Results......Page 97
6. Conclusion: Applications to HLT and HCI?......Page 99
Appendix: Other Tools, Schemes and Methods of Gesture Analysis......Page 100
References......Page 104
1. Introduction......Page 105
2. Study......Page 107
3. Data Analysis......Page 109
4. Results......Page 111
6. Related Work......Page 114
8. Conclusions......Page 116
Appendix: Questionnaire MYST III - EXILE......Page 118
References......Page 119
1. Introduction......Page 121
2. Breakdown and Repair......Page 125
3. Analysing Communicative Co-ordination......Page 133
4. Discussion......Page 134
References......Page 135
1. Context and Motivation......Page 138
2. Methodology and Experimental Set-Up......Page 141
3. Results: Presentation and Discussion......Page 148
4. Conclusion......Page 160
References......Page 161
1. Introduction......Page 165
2. Related Work......Page 166
3. Multimodal Multisensor Domain......Page 168
4. Results......Page 172
5. Single Multimodal Sensor Domain......Page 173
6. Integration......Page 181
References......Page 184
Part III: Animated Talking Heads and Evaluation......Page 187
1. Introduction......Page 188
2. Facial Animation and Visible Speech Synthesis......Page 189
3. Speech Science......Page 196
4. Language Learning......Page 199
5. Research on the Educational Impact of Animated Tutors......Page 202
6. Summary......Page 215
References......Page 216
1. Introduction......Page 220
2. The KTH Talking Head......Page 222
3. Effectiveness in Intelligibility and Information Presentation......Page 224
4. Effectiveness in Interaction......Page 228
5. Experimental Applications......Page 236
6. The Effective Agent as a Language Tutor......Page 240
7. Experiments and 3D Recordings for the Expressive Agent......Page 242
References......Page 244
1. Introduction......Page 249
2. Functions of Gaze......Page 252
3. The Experiment......Page 256
4. Discussion......Page 262
References......Page 264
Part IV: Architectures and Technologies for Advanced and Adaptive Multimodal Dialogue Systems......Page 267
1. Introduction......Page 268
3. MIND Overview......Page 270
4. Example Scenario......Page 271
5. Semantics-Based Representation......Page 273
6. Context-Based Multimodal Interpretation......Page 280
7. Discussion......Page 285
References......Page 286
1. Introduction......Page 289
2. An Intelligent Tutoring System for Damage Control......Page 290
3. An Architecture for Multimodal Dialogue Systems......Page 296
4. Activity Models......Page 297
5. Dialogue Management Architecture......Page 300
6. Benefits of ACI for Intelligent Tutoring Systems......Page 303
7. Conclusion......Page 304
References......Page 305
14 MIAMM – A Multimodal Dialogue System using Haptics......Page 308
1. Introduction......Page 309
2. Haptic Interaction in a Multimodal Dialogue System......Page 310
3. Visual Haptic Interaction – Concepts in MIAMM......Page 314
4. Dialogue Management......Page 320
5. The Multimodal Interface Language (MMIL)......Page 327
References......Page 332
1. Introduction......Page 334
2. Overview of Language Acquisition......Page 335
3. Dialogue Systems......Page 338
4. Language Knowledge Representation......Page 341
5. Dialogue Adaptation......Page 342
6. Experiments......Page 348
7. Conclusion......Page 352
References......Page 354
1. Introduction......Page 356
2. Modality Independent Dialogue Management......Page 358
3. Learning to Annotate Utterances......Page 363
4. Future work: Data Driven Dialogue Discovery......Page 367
5. Discussion......Page 368
References......Page 369
E......Page 372
M......Page 373
X......Page 374