Improving Surveys with Paradata: Analytic Uses of Process Information

IMPROVING SURVEYS WITH PARADATA......Page 3
CONTENTS......Page 7
PREFACE......Page 17
CONTRIBUTORS......Page 21
ACRONYMS......Page 23
1.1 Introduction......Page 25
1.2 Paradata and Metadata......Page 26
1.3 Auxiliary Data and Paradata......Page 27
1.4 Paradata in the Total Survey Error Framework......Page 28
1.6.1 Mode-Specific Paradata......Page 30
1.6.3 Quality of Paradata......Page 31
1.7 Future of Paradata......Page 32
References......Page 33
PART I PARADATA AND SURVEY ERRORS......Page 35
2.1 Introduction......Page 37
2.2.1 Call History Data......Page 38
2.2.2 Interviewer Observations......Page 41
2.2.3 Measures of the Interviewer–Householder Interaction......Page 43
2.3 Nonresponse Rates and Nonresponse Bias......Page 44
2.3.2 Call Records......Page 46
2.3.3 Interviewer Observations......Page 51
2.3.4 Observations of Interviewer–Householder Interactions......Page 53
2.4 Paradata and Responsive Designs......Page 54
2.5 Paradata and Nonresponse Adjustment......Page 55
2.6 Issues in Practice......Page 56
References......Page 58
3.1 Introduction......Page 67
3.2 Paradata and Measurement Error......Page 68
3.3.1 Time Stamps......Page 71
3.3.2 Keystrokes......Page 73
3.3.3 Mouse Clicks......Page 74
3.3.4 Behavior Codes......Page 75
3.3.5 Vocal Characteristics......Page 77
3.3.6 Interviewer Evaluations......Page 78
3.4.1 In-Person Surveys......Page 80
3.4.2 Telephone Surveys......Page 81
3.4.3 Web Surveys......Page 84
3.5.1 Paradata as Text Files......Page 85
3.5.3 Paradata as Variables......Page 88
References......Page 89
4.2 Review of Empirical Literature on the Use of Paradata for Measurement Error nvestigation......Page 97
4.2.1 Using Paradata to Understand the Question–Answering Process......Page 98
4.2.3 Reduction of Measurement Error......Page 99
4.3 Analyzing Paradata......Page 100
4.3.1 Units of Analysis......Page 101
4.3.2 Data Management......Page 102
4.3.4 Modeling Decisions......Page 106
4.4.1 Draisma and Dijkstra (2004)—Item Level: Paradata as an Independent Variable......Page 107
4.4.2 Malhotra (2008)—Survey Level: Paradata as an Independent Variable......Page 110
4.4.3 Yan and Tourangeau (2008)—Item Level: Paradata as a Dependent Variable......Page 111
4.4.4 Lenzner, Kaczmirek, and Lenzner (2010)—Survey Level: Paradata as a Dependent Variable......Page 112
4.5 Cautions......Page 113
References......Page 114
5.1 Introduction......Page 121
5.2.1 Postal Delivery Databases......Page 125
5.2.2 Housing Unit Listing......Page 128
5.2.3 Random Route Sampling......Page 130
5.2.4 Missed Unit Procedures......Page 132
5.3 Telephone Number Frames......Page 133
5.4 Household Rosters......Page 135
5.6 Subpopulation Frames......Page 137
5.7 Web Surveys......Page 138
Acknowledgments......Page 139
References......Page 140
PART II PARADATA IN SURVEY PRODUCTION......Page 145
6.1 Introduction......Page 147
6.2 From Repeated Cross-Section to Continuous Design......Page 148
6.3 Paradata Design......Page 153
6.4 Key Design Change 1: A New Employment Model......Page 158
6.5 Key Design Change 2: Field Efficient Sample Design......Page 159
6.6 Key Design Change 3: Replicate Sample Design......Page 161
6.7 Key Design Change 4: Responsive Design Sampling of Nonrespondents in a Second Phase......Page 163
6.8 Key Design Change 5: Active Responsive Design Interventions......Page 164
6.9 Concluding Remarks......Page 165
References......Page 167
7.1 Introduction......Page 169
7.2 Background......Page 170
7.3 The Survey Setting......Page 172
7.4 Experiments: Data and Methods......Page 173
7.4.1 Call Windows......Page 174
7.4.2 The Data......Page 175
7.4.3 The Models......Page 178
7.4.4 Procedure: Telephone Survey......Page 181
7.4.5 Procedure: Face-to-Face Survey......Page 184
7.5.1 Telephone Survey......Page 185
7.5.2 Face-to-Face Survey......Page 189
7.6 Discussion......Page 190
References......Page 193
8.1 Introduction......Page 195
8.2 Consent to Link Survey and Administrative Records......Page 199
8.2.1 Modeling Linkage Consent Using Paradata: Example from the Health and Retirement Study......Page 200
8.3 Consent to Collect Biomeasures in Population-Based Surveys......Page 202
8.3.1 Modeling Biomeasure Consent Using Paradata: Example from the Health and Retirement Study......Page 203
8.4 Switching Data Collection Modes......Page 204
8.4.1 Predicting Mode Switch Response Using Paradata: Example from a Survey of University Alumni......Page 205
8.4.2 Using Paradata for Intervention......Page 206
8.5 Income Item Nonresponse and Quality of Income Reports......Page 207
8.5.1 Studying Income Item Nonresponse and Quality of Income Reports Using Paradata: Examples from the Health and Retirement Study......Page 208
8.6 Summary......Page 210
References......Page 211
9.1 Introduction......Page 215
9.2 Defining and Choosing Key Performance Indicators (KPIs)......Page 217
9.3 KPI Displays and the Enduring Insight of Walter Shewhart......Page 224
9.3.1 Understanding a Process: Impediments to Clear Quality Control Steps......Page 229
9.3.2 Rules for Finding Special Cause Variation in a Control Chart......Page 231
9.4 Implementation Steps for Survey Analytic Quality Control with Paradata Control Charts......Page 234
9.5 Demonstrating a Method for Improving Measurement Process Quality Indicators......Page 237
9.6 Reflections on SPC, Visual Data Displays, and Challenges to Quality Control and Assurance with Survey Analytics......Page 244
9.7 Some Advice on Using Charts......Page 245
Appendix......Page 248
References......Page 250
10.1 Introduction......Page 255
10.2.1 What are Measurement Profiles?......Page 257
10.2.2 Measurement Profiles in the Labour Force Survey......Page 258
10.3 Tools for Monitoring Nonresponse and Measurement Profiles......Page 260
10.4.1 The Choice of Design Features in Monitoring Nonresponse and Measurement Profiles......Page 266
10.4.2 The Choice of Population Subgroups in Monitoring......Page 267
10.4.3 Partial R-indicators for the Sequential and Cumulative Analysis......Page 269
10.4.4 Adaptive Survey Designs......Page 274
10.5 Including Paradata Observations on Households and Persons......Page 277
10.6 General Discussion......Page 279
10.7 Take Home Messages......Page 280
References......Page 281
PART III SPECIAL CHALLENGES......Page 283
11.1 Survey Data Types......Page 285
11.3 Typology of Paradata in Web Surveys......Page 286
11.3.1 Uses of Paradata: Device Type......Page 288
11.3.2 Uses of Paradata: Questionnaire Navigation......Page 291
11.4 Using Paradata to Change the Survey in Real Time: Adaptive Scripting......Page 295
11.6 Software to Collect Paradata......Page 296
11.7 Analysis of Paradata: Levels of Aggregation......Page 297
11.8 Privacy and Ethical Issues in Collecting Web Survey Paradata......Page 298
11.9 Summary and Conclusions on Paradata in Web Surveys......Page 299
References......Page 300
12.1 Introduction......Page 305
12.2 Call Record Data......Page 307
12.3.1 Analysis Approaches and the Use of Multilevel Modeling......Page 309
12.3.2 Specifications of Multilevel Discrete-Time Event History Models for the Analysis of Call Record Data......Page 311
12.3.3 Modeling Strategy and Estimation of Models......Page 316
12.4.1 Analyzing Call Outcomes in the PASS Longitudinal Survey......Page 317
12.4.2 Analyzing Call Outcomes in the UK Census Nonresponse Link Study......Page 320
References......Page 328
13.1.1 Processes Under Control and Out of Control......Page 333
13.1.2 Motivating Example......Page 335
13.2.1 Definition......Page 337
13.2.2 Basis Functions......Page 339
13.2.3 Parameters of Interest......Page 340
13.2.4 Branching Splines......Page 342
13.2.5 Knot Density and Roughness......Page 343
13.3.1 Model Formulation......Page 345
13.3.2 Estimating Parameters......Page 346
13.3.3 Estimating the Function......Page 347
13.3.4 Difficulties with Likelihood Inference......Page 348
13.4.1 Bayesian Inference for the Smoothing Parameter......Page 349
13.4.2 Bayesian Intervals and Predictions......Page 350
13.5 Extensions......Page 353
A.1 Maximum-Likelihood Estimation......Page 354
A.2 Posterior Simulation......Page 357
A.3 Bayesian Inference About the Mean Function......Page 359
References......Page 361
14.1 Introduction......Page 363
14.2.1 Computer-Generated Process Data......Page 364
14.2.2 Interviewer-Recorded Call Record Data......Page 367
14.2.3 Interviewer Observations......Page 373
14.3.1 Computer-Generated Process Data......Page 376
14.3.2 Interviewer Observations......Page 377
References......Page 380
15.1 Introduction......Page 385
15.2.1 Simulation Parameters......Page 388
15.2.2 Alternative Estimators......Page 392
15.3.1 Scenario 1: D has a Positive Relationship with Y and a Negative Relationship with R, FNRs are Greater Than FPRs for Both Respondents and Nonrespondents, FNRs are Equal for Respondents and Nonrespondents, and FPRs are Equal for Respondents and Nonrespondents......Page 394
15.3.2 Scenario 2: D has a Positive Relationship with Both Y and R, FNRs are Greater Than FPRs for Both Respondents and Nonrespondents, FNRs are Equal for Respondents and Nonrespondents, and FPRs are Equal for Respondents and Nonrespondents......Page 397
15.3.3 Scenario 3: D has a Positive Relationship with Y and a Negative Relationship with R, FNRs are Greater Than FPRs for Both Respondents and Nonrespondents, FNRs for Nonrespondents are Greater Than Those for Respondents, and FPRs for Nonrespondents are Greater Than Those for Respondents......Page 400
15.3.4 Scenario 4: D has a Positive Relationship with Both Y and R, FNRs are Greater Than FPRs for Both Respondents and Nonrespondents, FNRs for Nonrespondents are Greater Than Those for Respondents, and FPRs for Nonrespondents are Greater Than Those for Respondents......Page 402
15.3.5 Additional Scenarios......Page 403
15.4 Take Home Messages......Page 407
15.5 Future Research......Page 409
References......Page 411
INDEX......Page 413
WILEY SERIES......Page 417