Measuring the user experience: collecting, analyzing, and presenting usability metrics

Front Cover......Page 1
Measuring the User Experience......Page 4
Copyright Page......Page 5
Dedication......Page 6
Contents......Page 8
Preface to the Second Edition......Page 14
Acknowledgments......Page 16
Biographies......Page 18
1 Introduction......Page 20
1.1 What is User Experience......Page 23
1.2 What are User Experience Metrics?......Page 25
1.3 The Value of UX Metrics......Page 27
1.4 Metrics for Everyone......Page 28
1.5 New Technologies in UX Metrics......Page 29
Myth 1: Metrics Take Too Much Time to Collect......Page 30
Myth 5: UX Metrics are Too Noisy......Page 31
Myth 8: No Metrics Exist for the Type of Issues We are Dealing with......Page 32
Myth 10: It’s Difficult to Collect Reliable Data with a Small Sample Size......Page 33
2 Background......Page 34
2.2.1 Nominal Data......Page 35
2.2.2 Ordinal Data......Page 36
2.2.3 Interval Data......Page 37
2.3.1 Measures of Central Tendency......Page 38
2.3.2 Measures of Variability......Page 40
2.3.3 Confidence Intervals......Page 41
2.3.4 Displaying Confidence Intervals as Error Bars......Page 43
2.4 Comparing Means......Page 44
2.4.1 Independent Samples......Page 45
2.4.2 Paired Samples......Page 46
2.4.3 Comparing More Than Two Samples......Page 48
2.5.1 Correlations......Page 49
2.6.1 The χ2 Test......Page 50
2.7 Presenting your Data Graphically......Page 51
2.7.1 Column or Bar Graphs......Page 52
2.7.2 Line Graphs......Page 54
2.7.3 Scatterplots......Page 55
2.7.4 Pie or Donut Charts......Page 57
2.7.5 Stacked Bar or Column Graphs......Page 58
2.8 Summary......Page 59
3 Planning......Page 60
3.1.1 Formative Usability......Page 61
3.1.2 Summative Usability......Page 62
3.2.2 Satisfaction......Page 63
3.3.1 Completing a Transaction......Page 64
3.3.3 Evaluating Frequent Use of the Same Product......Page 66
3.3.5 Increasing Awareness......Page 67
3.3.6 Problem Discovery......Page 68
3.3.7 Maximizing Usability for a Critical Product......Page 69
3.3.9 Evaluating the Impact of Subtle Changes......Page 70
3.4 Evaluation Methods......Page 71
3.4.1 Traditional (Moderated) Usability Tests......Page 72
3.4.2 Online (Unmoderated) Usability Tests......Page 73
3.4.3 Online Surveys......Page 75
3.5.1 Budgets and Timelines......Page 76
3.5.2 Participants......Page 77
3.5.4 Data Cleanup......Page 79
3.6 Summary......Page 80
4 Performance Metrics......Page 82
4.1 Task Success......Page 84
4.1.1 Binary Success......Page 85
Calculating Confidence Intervals for Binary Success......Page 88
4.1.2 Levels of Success......Page 89
How to Collect and Measure Levels of Success......Page 90
4.1.3 Issues in Measuring Success......Page 92
4.2 Time on Task......Page 93
4.2.2 How to Collect and Measure Time on Task......Page 94
Tabulating Time Data......Page 95
4.2.3 Analyzing and Presenting Time-on-Task Data......Page 97
Thresholds......Page 98
Distributions and Outliers......Page 99
Using a Concurrent Think-Aloud Protocol......Page 100
4.3.1 When to Measure Errors......Page 101
4.3.2 What Constitutes an Error?......Page 102
4.3.4 Analyzing and Presenting Errors......Page 103
4.4 Efficiency......Page 105
4.4.1 Collecting and Measuring Efficiency......Page 106
4.4.2 Analyzing and Presenting Efficiency Data......Page 107
Lostness......Page 108
4.4.3 Efficiency as a Combination of Task Success and Time......Page 109
4.5 Learnability......Page 111
4.5.2 Analyzing and Presenting Learnability Data......Page 113
4.6 Summary......Page 115
5 Issue-Based Metrics......Page 118
5.1 What is a Usability Issue?......Page 119
5.1.1 Real Issues versus False Issues......Page 120
5.2.1 In-Person Studies......Page 121
5.3 Severity Ratings......Page 122
5.3.1 Severity Ratings Based on the User Experience......Page 123
5.3.2 Severity Ratings Based on a Combination of Factors......Page 124
5.3.3 Using a Severity Rating System......Page 125
5.4 Analyzing and Reporting Metrics for Usability Issues......Page 126
5.4.1 Frequency of Unique Issues......Page 127
5.4.3 Frequency of Participants......Page 128
5.4.4 Issues by Category......Page 129
5.5 Consistency in Identifying Usability Issues......Page 130
5.6 Bias in Identifying Usability Issues......Page 132
5.7.1 Five Participants is Enough......Page 134
5.7.2 Five Participants is Not Enough......Page 136
5.7.3 Our Recommendation......Page 137
5.8 Summary......Page 138
6 Self-Reported Metrics......Page 140
6.2.1 Likert Scales......Page 142
6.2.2 Semantic Differential Scales......Page 143
6.2.4 How to Collect Ratings......Page 144
6.2.6 General Guidelines for Rating Scales......Page 145
6.2.7 Analyzing Rating-Scale Data......Page 146
6.3.1 Ease of Use......Page 150
6.3.3 Expectation Measure......Page 151
6.3.4 A Comparison of Post-task Self-Reported Metrics......Page 152
6.4.2 System Usability Scale......Page 156
6.4.3 Computer System Usability Questionnaire......Page 159
6.4.4 Questionnaire for User Interface Satisfaction......Page 160
6.4.5 Usefulness, Satisfaction, and Ease-of-Use Questionnaire......Page 161
6.4.6 Product Reaction Cards......Page 163
6.4.7 A Comparison of Postsession Self-Reported Metrics......Page 164
6.4.8 Net Promoter Score......Page 165
6.6 Online Services......Page 166
6.6.2 American Customer Satisfaction Index......Page 167
6.6.3 OpinionLab......Page 168
6.6.4 Issues with Live-Site Surveys......Page 171
6.7.1 Assessing Specific Attributes......Page 173
6.7.2 Assessing Specific Elements......Page 175
6.7.3 Open-Ended Questions......Page 177
6.7.4 Awareness and Comprehension......Page 178
6.7.5 Awareness and Usefulness Gaps......Page 179
6.8 Summary......Page 180
7.1 Observing and Coding Unprompted Verbal Expressions......Page 182
7.2.1 How Eye Tracking Works......Page 184
7.2.2 Visualizing Eye-Tracking Data......Page 186
7.2.3 Areas of Interest......Page 189
7.2.4 Common Eye-Tracking Metrics......Page 191
Time to First Fixation......Page 192
7.2.5 Eye-Tracking Analysis Tips......Page 193
7.2.6 Pupillary Response......Page 194
7.3.1 Affectiva and the Q-Sensor......Page 195
7.3.2 Blue Bubble Lab and Emovision......Page 198
7.3.3 Seren and Emotiv......Page 199
7.4.1 Heart Rate Variance......Page 201
7.4.3 Other Measures......Page 202
7.5 Summary......Page 204
8.1 Single Usability Scores......Page 206
8.1.1 Combining Metrics Based on Target Goals......Page 207
8.1.2 Combining Metrics Based on Percentages......Page 208
8.1.3 Combining Metrics Based on Z Scores......Page 215
8.1.4 Using Single Usability Metric......Page 217
8.2 Usability Scorecards......Page 219
8.3.1 Comparison to Goals......Page 223
8.3.2 Comparison to Expert Performance......Page 225
8.4 Summary......Page 227
9.1 Live Website Data......Page 228
9.1.1 Basic Web Analytics......Page 229
9.1.2 Click-Through Rates......Page 232
9.1.3 Drop-Off Rates......Page 234
9.1.4 A/B Tests......Page 235
9.2 Card-Sorting Data......Page 237
9.2.1 Analyses of Open Card-Sort Data......Page 238
Hierarchical Cluster Analysis......Page 240
Multidimensional Scaling......Page 241
9.2.2 Analyses of Closed Card-Sort Data......Page 243
9.2.3 Tree Testing......Page 246
9.3 Accessibility Data......Page 247
9.4 Return-On-Investment Data......Page 251
9.5 Summary......Page 255
10 Case Studies......Page 256
10.1 Net Promoter Scores and the Value of a Good User Experience......Page 257
10.1.1 Methods......Page 258
10.1.2 Results......Page 259
10.1.3 Prioritizing Investments in Interface Design......Page 260
10.1.4 Discussion......Page 261
10.1.5 Conclusion......Page 262
10.2 Measuring the Effect of Feedback on Fingerprint Capture......Page 263
10.2.1 Methodology......Page 264
10.2.2 Discussion......Page 271
References......Page 272
10.3 Redesign of a Web Experience Management System......Page 273
10.3.1 Test Iterations......Page 274
10.3.2 Data Collection......Page 275
10.3.3 Workflow......Page 276
10.3.4 Results......Page 280
Biographies......Page 281
10.4 Using Metrics to Help Improve a University Prospectus......Page 282
10.4.1 Example 1: Deciding on Actions after Usability Testing......Page 283
10.4.2 Example 2: Site-Tracking Data......Page 286
10.4.3 Example 3: Triangulation for Iteration of Personas......Page 288
Biographies......Page 289
10.5.1 Background......Page 290
10.5.2 Methods......Page 291
10.5.3 Biometric Findings......Page 292
10.5.4 Qualitative Findings......Page 293
10.5.5 Conclusions and Practitioner Take-Aways......Page 294
References......Page 295
Biographies......Page 296
11.1 Make Data Come Alive......Page 298
11.2 Don’t Wait to be Asked to Measure......Page 300
11.4 Plan Early......Page 301
11.5 Benchmark Your Products......Page 302
11.6 Explore Your Data......Page 303
11.8 Show Your Confidence......Page 304
11.9 Don’t Misuse Metrics......Page 305
11.10 Simplify Your Presentation......Page 306
References......Page 308
Index......Page 316