Mastering Voice Interface

Table of Contents
About the Authors
About the Technical Reviewers
Acknowledgments
Introduction
Part I: Conversational Voice System Foundations
	Chapter 1: Say Hello to Voice Systems
		Voice-First, Voice-Only, and Conversational Everything
		Claims About Voice Technology
			Claim: Everyone Has a Smart Voice Device and Uses It All the Time
			Claim: You Can Simply “Add Voice” to an Existing GUI or Touch Interface
			Claim: Voice or Chatbot, Both Are Conversational So Basically the Same
			Claim: I Speak the Language; What More Is There to VUI Design?
			Claim: Every Voice Solution Needs a Strong Personality
			Claim: Hire a Scriptwriter; They Can Write Conversations
			Claim: Recognition Is a Solved Problem; It’s Basically Perfect Today
			Claim: AI Takes Care of Understanding What People Say
			Claim: IVRs Are Irrelevant Today, Nothing to Be Learned from Them
			Claim: That Ship Has Sailed; Alexa Is the Winner
			Claim: Everyone Needs a Voice Solution; Voice Is Great for Everything
		Introduction to Voice Technology Components
			Speech-to-Text
				In-Depth: Is STT Better Than Human Recognition and Understanding?
			Natural Language Understanding
			Dialog Management
			Natural Language Generation
			Text-to-Speech
				In-Depth: TTS Synthesis—Is It Like Human Speech?
		The House That Voice Built: The Phases of Voice Development Success
			Plan
			Design
			Build
			Test
			Deploy and Assess
			Iterate
		What Next?
		Summary
	Chapter 2: Keeping Voice in Mind
		Why Voice Is Different
		Hands-On: A Precoding Thought Experiment
		Voice Dialog and Its Participants
			The Human: Spoken Natural Language
			The Computer: Voice Recognition and Interpretation
			Human-Computer Voice Dialog
		What’s Next?
		Summary
	Chapter 3: Running a Voice App—and Noticing Issues
		Hands-On: Preparing the Restaurant Finder
		Choosing Voice Platforms
		Hands-On: Implementing the Restaurant Finder
			Basic Setup
			Step 1: Respond to an Invocation
			Step 2: Specify What the User Says
			Step 3: Specify What the VUI Says
			Step 4: Connect Dialogflow to Actions on Google
			Step 5: Test Your VUI
				Testing Using Dialogflow
				Testing Using the Actions console
				Testing with Assistant, Home, or Nest Devices
			Step 6: Save, Export, or Import Your Work
		Why We’re Using Actions on Google and Assistant
		Google’s Voice Development Ecosystem
			Actions on Google
				Dialogflow
		The Pros and Cons of Relying on Tools
		Hands-On: Making Changes
			Adding More Phrases with the Same Meaning
				Pushing the Limits
				Rejection
				Fixing Rejection
			Expanding Information Coverage
			Adding Granularity
				Preview: Static vs. Dynamic Responses
		What’s Next?
		Summary
Part II: The Planning Phase: Requirements Discovery and High-Level Design Definition
	Chapter 4: Define Through Discovery: Building What, How, and Why for Whom
		General Functionality: What Are You Building?
		User Needs: Who’ll Use It and What Do They Want?
			Demographics and Characteristics
			Engagement Patterns
			Mental Models and Domain Knowledge
			Environment and State of Mind
		From General to Detailed Functionality
			Step 1: Define the Core Features
			Step 2: From Features to Intents and Slots
				Step 3: Identify Potential Complications
		Business Requirements: Why Are You Building It?
			General Purpose and Goal
				Primary and Secondary Goals
				Success Metrics
				Domain, Device, and Other Modalities
			Underlying Service and Existing Automation
			Branding and Terminology
				Prompt Format: Text-to-Speech or Recordings
				Terminology and Pronunciation
				Data Needs
			Access and Availability
				Privacy and Security Concerns
				Access Limitations Based on Time or Place
				Special Announcements
			Legal and Business Constraints
				Compliance
				Identification and Authentication
				Error Handling and Navigation Behaviors
		System Requirements: How Will You Build It?
			Recognizer, Parser, and Interpreter
			External Data Sources
			Data Storage and Data Access
			Other System Concerns
		What’s Next?
		Summary
	Chapter 5: From Discovery to UX and UI: Tools of Voice Design
		Where to Find Early User Data on Any Budget
			Online Research and Crowdsourcing Options
				Blogs and Articles
				Industry Studies
				Customer Reviews
				Supporting Materials
				Crowdsourced Data
			Dialog Participant Observation
				Actual or Potential Primary User Observation
				Secondary Dialog Participant Observation
				Expert Reviews and Heuristic Evaluation
			Focus Groups, Interviews, and Surveys
				In-Depth Interviews and Focus Groups
				Subject Expert Panel
				Targeted Surveys
				In-App Data Collection
		How Discovery Informs VUI Design Decisions
			Discovery Informing Design: An Example
			Dialog Manager Graph, Yes; Hierarchical Decision Tree, No
		Capturing and Documenting VUI Design
			Dialog Flows
			Sample Dialogs
			Detailed Design Specifications
			VUI Design Documentation Approaches
				Self-Contained Design Tools
				Open Platform-Specific Tools
				Proprietary or Closed Platform-Specific Tools
				Standard Documentation Tools
			Prototyping and Testing Your Assumptions
			Voice UX and Prototyping Approaches
				Hallway “Guerilla” Studies
				Low-Fidelity Mockups (Voice Version of “Wireframes”)
				Wizard of Oz (“WOZ”)
				Functional Prototypes
		What’s Next?
		Summary
Part III: The Building Phase: Design-Informed Development
	Chapter 6: Applying Human “Rules of Dialog” to Reach Conversation Resolution
		Dialog Acts, Games, and Turns—and Grice
			In-Depth: Reaching the End of a Dialog Through Grice’s Cooperative Principle
		Question Answering
			Building Out the Question Answering Example
			In-Depth: Entities, Parameters, and Slots
			Design Notes on Question Answering
		Action Requests
			Building Action Requests
			Design Notes on Action Requests
		Task Completion Requests
			Building Task Completion Example 1
			Building Task Completion Example 2
			Design Notes on Task Completion
		Fully Specified Requests
			Single-Slot Requests
				Implementing Single-Slot Requests
				Design Notes on Single-Slot Requests
			Multi-slot Requests
				Implementing Multi-slot Requests
				Design Notes on Multi-slot Requests
		Determining Dialog Acts Based on Feature Discovery
		Dialog Completion
			Responding to “Goodbye” and “Thanks”
				Implementing the End of a Conversation
		What’s Next?
		Summary
	Chapter 7: Resolving Incomplete Requests Through Disambiguation
		Incomplete Requests
			Reaching Completeness Through Dialog Management
				Implementing Request Completion
				Notes on Implementing Request Completion
		Ambiguous Requests
		Disambiguation Methods
			Logic-Based Assumptions
			Yes/No Questions
				Implementing Yes/No Questions
				Notes on Implementing Yes/No Questions
			A/B Sets
				Implementing A/B Sets
				Notes on Implementing A/B Sets
			Static Lists
				Implementing Static Lists
				Notes on Implementing Static Lists
			Dynamic Lists
				Implementing Dynamic Lists
				Notes on Implementing Dynamic Lists
			Open Sets
				Implementing Open Sets
				Notes on Implementing Open Sets
			Menus
				Implementing Menus
				Notes on Implementing Menus
		Testing on the Device to Find and Solve Issues
			Two Big Lessons
		Webhooks 1: Toward Code Independence
			Fulfillment and Webhooks
			Webhook Overview
			Webhook in Depth
			Contexts, Context Parameters, and Follow-Up Intents
		What’s Next
		Summary
	Chapter 8: Conveying Reassurance with Confidence and Confirmation
		Conveying Reassurance and Shared Certainty
			Setting Expectations with Your Implications
		Webhooks 2
			JSON
			The Webhook Request
			The Webhook Response
				Custom Payloads
			Implementing the Webhook
		Confirmation Methods
			Non-verbal Confirmation
				Code for Non-verbal Confirmation
				Notes on Non-verbal Confirmation
			Generic Acknowledgment
			Implicit Confirmation
				Code for Implicit Confirmation
				What If the User Says No?
				Notes on Implicit Confirmation
			Explicit Confirmation
				Code for Explicit Confirmation
				Dialogflow, App Engine, and Statelessness
				Notes on Explicit Confirmation
		Confirmation Placement: Slots vs. Intents
		Disconfirmation: Dealing with “No”
			Code for Disconfirmation
			Notes on Disconfirmation
		Additional Reassurance Techniques and Pitfalls
			System Pronunciation
			Backchannels
			Discourse Markers
			VUI Architecture
		Choosing the Right Reassurance Method
		Summary
	Chapter 9: Helping Users Succeed Through Consistency
		Universals
			Providing Clarification and Additional Information
			Providing a Do-Over
			Providing an Exit
				Coding Universals
				Step 1: Creating Intents
				Step 2: Coding the Webhook
				Step 3: Adding Repeat
				Step 4: Checking Context Robustness
				Step 5: Ending the Conversation
				Sidebar: Cleaning Up the Code
		Navigation
			Landmarks
			Non-verbal Audio
				Code for NVA
			Content Playback Navigation
			List Navigation
		Consistency, Variation, and Randomization
			Code for Randomization
			Working with Built-In Global Intents
			Consistency and Standards Across VUIs and Frameworks
		What’s Next?
		Summary
	Chapter 10: Creating Robust Coverage for Speech-to-Text Resolution
		Recognition Is Speech-to-Text Interpretation
		Inside the STT Box
			Creating Sentences with Smaller Pieces
			Using STT to Build Up Sentences
		Recognition Engines
		Grammar Concepts
			Coverage
			Recognition Space
			Static or Dynamic, Large or Small
			End-Pointing
			Multiple Hypotheses
		Types of Grammars
			Rule-Based Grammars
			Statistical Models
			Hot Words, Wake Words, and Invocations
		Working with Grammars
			Writing Rule-Based Regular Expressions
				Regular Expressions Defined
				Grammars and Training Phrases
				Sub-grammars
				Big Sub-grammars
				Pronunciations
		How to Succeed with Grammars
			Bootstrap
			Normalize Punctuation and Spellings
			Handle Unusual Pronunciations
			Use Domain Knowledge
			Understand the Strengths and Limitations of STT
		Hands-On: Taking Control over Coverage
			Changes in Dialogflow
				Remove Existing Sample Phrases
				Add an Event
			Regular Expressions in the Webhook
				Handling the Phrase If No Other Intent Does
				Why Do It This Way?
				Programming Sidebar: Debugging the Webhook
		Limitations on Grammar Creation and Use
		What’s Next
		Summary
	Chapter 11: Reaching Understanding
		From Words to Meaning
			NLP
			NLU
				In-Depth: What’s “Smart,” AI, and Understanding?
		Parsing
		Machine Learning and NLU
		Ontologies, Knowledge Bases, and Content Databases
		Intents
			Intent Tagging and Tagging Guides
			Middle Layers: Semantic Tags vs. System Endpoints
		Putting It All Together
			Matching Wide or Narrow
			Multiple Grammars, Multiple Passes
			The Stanford Parser Revisited
			Determining Intent
				Introducing spaCy
				Loading spaCy
				Calling spaCy
				Using spaCy’s Dependency Parse
				Dialogflow Changes
		In-Depth: Understanding, Parsing, and Knowledge in Practice
			Revisiting Intents
			Machine Learning and Using Knowledge
			The Not-So-Distant Future?
		What’s Next?
		Summary
	Chapter 12: Applying Accuracy Strategies to Avoid Misunderstanding
		Accuracy Robustness Concepts
		Accuracy Robustness Strategies
			Examples
				Implementing Examples
			Providing Help
				Notes on Designing for Help
			Just-in-Time Information
				Notes on Designing for JIT
			Hidden Options and “None of Those”
			Recognize-and-Reject
			One-Step Correction
				Implementing One-Step Correction
				Notes on One-Step Correction
			Tutorials
			Spelling
			Narrowing the Recognition Space
		Advanced Techniques
			Multi-tier Behavior and Confidence Scores
				Implementing Multi-tier Confirmation
				Notes on Multi-tier Confidence-Based Design
			N-Best and Skip Lists
				Implementing Skip Lists
				Notes on N-Best and Skip List Design
			Probabilities
			Timeout and Latency
		What’s Next
		Summary
	Chapter 13: Choosing Strategies to Recover from Miscommunication
		Recovery from What?
			Recognition, Intent, or Fulfillment Errors
		Recovery Strategies
			Meaningful Contextual Prompts
			Escalating Prompts
				Code for Escalating Prompts
			Tapered Prompts
			Rapid Reprompt
			Backoff Strategies
		When to Stop Trying
			Max Error Counts
				Code for Keeping Track of Max Errors
			Transfers
			Choosing a Recovery Strategy
		What’s Next
		Summary
	Chapter 14: Using Context and Data to Create Smarter Conversations
		Why There’s No Conversation Without Context
			Hands-On: Context Complexity Thought Experiment
		Data Access
			External Accounts and Services
		Persistence Tracking
		Context-Aware and Context-Dependent Dialogs
			Discourse Markers and Acknowledgments
				Code for Context-Aware Dialogs
			Anaphora Resolution
		Follow-Up Dialogs and Linked Requests
			Code for Follow-Up Dialogs
			Proactive Behaviors
			Code for Proactive Behaviors
			Topic, Domain, and World Knowledge
			Geolocation-Based Behavior
			Proximity and Relevance
			Number and Type of Devices
			Time and Day
			User Identity, Preferences, and Account Types
			User Utterance Wording
			System Conditions
		Tracking Context in Modular and Multi-turn Dialogs
			Fulfillment
		What’s Next?
		Summary
	Chapter 15: Creating Secure Personalized Experiences
		The Importance of Knowing Who’s Talking
		Individualized Targeted Behaviors
			Concepts in Personalization and Customization
			Implementing Individualized Experiences
				Code for Personalized Experiences
				In-Depth: Context or History?
		Authorized Secure Access
			Approaches to Identification and Authentication
			Implementing Secure Gated Access
				Notes on Implementing Access
		Privacy and Security Concerns
		System Persona
			Defining and Implementing a System Persona
			How Persona Affects Dialogs
		System Voice Audio
			TTS Synthesis or Voice Talent, Generated or Recorded
			Finding and Working with Voice Talents
			One or Several Voices
			Prompt Management
		Emotion and Style
		Voice for Specific User Groups
		What’s Next?
		Summary
Part IV: The Launch Phase: Verifying and Tuning
	Chapter 16: Testing and Measuring Voice System Performance
		Testing Voice System Performance
		Recognition Testing
			Dialog Traversal: Functional End-to-End Testing
			Wake Word and Speech Detection Testing
			Other Types of System Integration Testing
		Testing Usability and Task Completion
			Voice Usability Testing Concepts
				Study Methodology: Task Scenarios and Surveys
				Study Environment: In-Lab or Remote
				Study Administration: Moderated or Unmoderated
				Results: Quantitative or Qualitative
			Wizard of Oz Studies
				Developing the WOZ
		Tracking and Measuring Performance
			Recognition Performance Metrics
			Task Completion Metrics
			Audio and TTS Testing
			User Satisfaction Metrics
				Functional System Task Scenario Studies
				Administered and Crowdsourced Surveys
				In-App Automated Surveys
				Business Metrics
		What’s Next?
		Summary
	Chapter 17: Tuning and Deploying Voice Systems
		Tuning: What Is It and Why Do You Do It?
			Why Recognition Accuracy Isn’t Enough
			Analyzing Causes of Poor System Performance
		Tuning Types and Approaches
			Log-Based vs. Transcription-Based Tuning
			Coverage Tuning
				Ambiguity Tuning
			Recognition Accuracy Tuning
				Acoustic Confusability Tuning
				Dictionary Tuning
				Confidence Threshold
				Timeout and Probability Parameters
				In-Depth: Building Accuracy from Sample Phrases
			Finding and Using Recognition Accuracy Data
			Task Completion Tuning
				Code for Task Completion Analysis
			Dialog Tuning
			Intent Tuning
			How to Prioritize Your Tuning Efforts
		Mapping Observations to the Right Remedy
			Reporting On and Using Tuning Results
		How to Maximize Deployment Success
			Know When to Tune
			Understand Tuning Complexities to Avoid Pitfalls
		What’s Next?
		Summary
Index