Evaluation: A Systematic Approach

Cover......Page 1
EVALUATION......Page 2
CONTENTS......Page 6
PREFACE......Page 10
KEY CONCEPTS FOR CHAPTER 1......Page 13
CHAPTER 1 PROGRAMS, POLICIES, AND EVALUATIONS......Page 14
WHAT IS EVALUATION RESEARCH?......Page 15
A BRIEF HISTORY OF EVALUATION RESEARCH......Page 20
Evaluation Research as a Social Science Activity......Page 21
The Boom Period in Evaluation Research......Page 22
The Emergence of Government Programs......Page 24
The Development of Policy and Public Administration Specialists......Page 25
The Great Society and Its Aftermath: Political Ideology and the Evaluation Enterprise......Page 26
The Evaluation Field in the 1990s......Page 30
Application of Social Research Procedures......Page 31
The Effectiveness of Social Intervention Programs......Page 33
Adapting Evaluation to the Political and Organizational Context of the Program......Page 35
Informing Social Action to Improve Social Conditions......Page 36
EVALUATION RESEARCH IN PRACTICE......Page 38
Evaluation and the Volatility of Social Programs......Page 39
Scientific Versus Pragmatic Evaluation Postures......Page 40
Diversity in Evaluation Outlooks and Approaches......Page 43
WHO CAN DO EVALUATIONS?......Page 44
SUMMARY......Page 46
KEY CONCEPTS FOR CHAPTER 2......Page 47
CHAPTER 2 TAILORING EVALUATIONS......Page 48
WHAT ASPECTS OF THE EVALUATION PLAN MUST BE TAILORED?......Page 49
The Purposes of the Evaluation......Page 50
The Program Structure and Circumstances......Page 54
The Stage of Program Development......Page 55
The Administrative and Political Context of the Program......Page 58
The Conceptual and Organizational Structure of the Program......Page 61
The Resources Available for the Evaluation......Page 63
THE NATURE OF THE EVALUATOR-STAKEHOLDER RELATIONSHIP......Page 65
EVALUATION QUESTIONS AND EVALUATION METHODS......Page 73
Assessment of Program Theory......Page 75
Assessment of Program Process......Page 78
Impact Assessment......Page 81
Efficiency Assessment......Page 83
STITCHING IT ALL TOGETHER......Page 85
SUMMARY......Page 87
KEY CONCEPTS FOR CHAPTER 3......Page 89
CHAPTER 3 IDENTIFYING ISSUES AND FORMULATING QUESTIONS......Page 90
WHAT MAKES A GOOD EVALUATION QUESTION?......Page 92
Evaluation Questions Must Be Reasonable and Appropriate......Page 93
Evaluation Questions Must Be Answerable......Page 94
Criteria for Program Performance......Page 95
Typical Evaluation Questions......Page 98
DETERMINING THE QUESTIONS ON WHICH THE EVALUATION SHOULD FOCUS......Page 99
Representing the Concerns of the Evaluation Sponsor and Major Stakeholders......Page 101
Obtaining Input From Stakeholders......Page 103
Topics for Discussion With Stakeholders......Page 105
Analysis of Program Assumptions and Theory......Page 109
The Program Impact Theory......Page 113
The Program Service Utilization Plan......Page 120
The Program's Organizational Plan......Page 122
How to Articulate Program Theory......Page 124
COLLATING EVALUATION QUESTIONS AND SETTING PRIORITIES......Page 126
SUMMARY......Page 127
KEY CONCEPTS FOR CHAPTER 4......Page 129
CHAPTER 4 ASSESSING THE NEED FOR A PROGRAM......Page 130
THE ROLE OF EVALUATORS IN DIAGNOSING SOCIAL CONDITIONS AND SERVICE NEEDS......Page 131
DEFINING SOCIAL PROBLEMS......Page 136
SPECIFYING THE EXTENT OF THE PROBLEM: WHEN, WHERE, AND HOW BIG?......Page 137
Using Existing Data Sources to Develop Estimates......Page 138
Using Social Indicators to Identify Trends......Page 141
Estimating Problem Parameters Through Social Research......Page 142
Surveys and Censuses......Page 143
Forecasting Needs......Page 146
DEFINING AND IDENTIFYING THE TARGETS OF INTERVENTIONS......Page 148
Direct and Indirect Targets......Page 149
Target Boundaries......Page 150
Varying Perspectives on Target Specification......Page 153
Incidence and Prevalence......Page 154
Need and Demand......Page 155
Rates......Page 156
DESCRIBING THE NATURE OF SERVICE NEEDS......Page 157
Qualitative Methods for Describing Needs......Page 158
SUMMARY......Page 162
KEY CONCEPTS FOR CHAPTER 5......Page 165
CHAPTER 5 EXPRESSING AND ASSESSING PROGRAM THEORY......Page 166
THE EVALUABILITY ASSESSMENT PERSPECTIVE......Page 168
ELICITING AND EXPRESSING PROGRAM THEORY......Page 171
What Is a Program for Purposes of Program Theory?......Page 172
The Implicit Theory of Program Personnel and Other Stakeholders......Page 173
Topics for Attention During Document Review, Interviews, and Observations......Page 177
Frameworks for Assessing Program Theory......Page 184
Assessment in Relation to Social Needs......Page 185
Assessment of Logic and Plausibility......Page 188
Assessment Through Comparison With Research and Practice......Page 191
Assessment Via Preliminary Observation......Page 194
Outcomes and Responses to the Results of Program Theory Assessment......Page 197
SUMMARY......Page 198
KEY CONCEPTS FOR CHAPTER 6......Page 201
CHAPTER 6 MONITORING PROGRAM PROCESS AND PERFORMANCE......Page 202
WHAT IS PROGRAM MONITORING?......Page 203
Process or Implementation Evaluation......Page 208
Routine Program Monitoring and Management Information Systems......Page 210
Performance Measurement and Monitoring......Page 212
Monitoring From the Evaluator's Perspective......Page 214
Monitoring From an Accountability Perspective......Page 215
Monitoring From a Management Perspective......Page 216
MONITORING SERVICE UTILIZATION......Page 218
Coverage and Bias......Page 219
Measuring and Monitoring Coverage......Page 220
Program Records......Page 222
Surveys......Page 224
MONITORING ORGANIZATIONAL FUNCTIONS......Page 225
"Nonprograms" and Incomplete Intervention......Page 226
Unstandardized Intervention......Page 228
Specification of Services......Page 229
Program Support Functions......Page 230
MONITORING PROGRAM OUTCOMES......Page 231
Guidelines for Outcome Indicators......Page 233
Pitfalls in Outcome Monitoring......Page 234
Data Collected by the Evaluator......Page 236
Service Record Data......Page 237
Management Information Systems......Page 239
ANALYSIS OF MONITORING DATA......Page 240
Conformity of the Program to Its Design......Page 241
SUMMARY......Page 242
KEY CONCEPTS FOR CHAPTER 7......Page 245
CHAPTER 7 STRATEGIES FOR IMPACT ASSESSMENT......Page 246
The Experimental Model......Page 247
Linking Interventions to Outcomes......Page 249
"Perfect" Versus "Good Enough" Impact Assessments......Page 250
Gross Versus Net Outcomes......Page 251
Uncontrolled Selection......Page 252
Endogenous Change......Page 253
Maturational Trends......Page 254
Stochastic Effects......Page 255
Measurement Reliability......Page 258
Measurement Validity......Page 260
Choice of Outcome Measures......Page 261
The Hawthorne Effect and Other Delivery System Contaminants......Page 265
Missing Information......Page 266
DESIGN STRATEGIES FOR ISOLATING THE EFFECTS OF EXTRANEOUS FACTORS......Page 268
Full- Versus Partial-Coverage Programs......Page 269
Design IA: Randomized Experiments......Page 271
Design IB: Quasi-Experiments......Page 274
Design IIA: Simple Before-and-After Studies......Page 277
Design IID: Time-Series Analyses (many repeated measures)......Page 278
JUDGMENTAL APPROACHES TO IMPACT ASSESSMENT......Page 279
QUANTITATIVE VERSUS QUALITATIVE DATA IN IMPACT ASSESSMENTS......Page 280
Reproducibility......Page 282
Generalizability......Page 283
Pooling Evaluations: Meta-Analysis......Page 284
CHOOSING THE RIGHT IMPACT ASSESSMENT STRATEGY......Page 285
SUMMARY......Page 286
KEY CONCEPTS FOR CHAPTER 8......Page 289
UNITS OF ANALYSIS......Page 290
The Concept of Control and Experimental Groups......Page 291
Implementing Control Group Evaluations......Page 292
Using Randomization to Establish Comparability......Page 294
The Logic of Randomized Experiments......Page 295
Examples of Randomized Experiments in Impact Assessment......Page 296
Near Experiments: Conditions of "Ignorability"......Page 301
Simple Randomized Experiments......Page 303
Complex Randomized Experiments......Page 304
Programs in Early Stages of Implementation......Page 308
Ethical Considerations......Page 312
Differences Between Experimental and Actual Intervention Delivery......Page 313
Integrity of Experiments......Page 314
SUMMARY......Page 316
KEY CONCEPTS FOR CHAPTER 9......Page 319
QUASI-EXPERIMENTAL IMPACT ASSESSMENT......Page 320
Measuring Impacts in Quasi-Experimental Evaluations......Page 322
Ex Post Quasi-Experiments......Page 323
Constructing Control Groups by Matching......Page 324
Identifying Characteristics for Matching......Page 326
Matching Procedures......Page 328
The Logic Behind Statistical Controls......Page 331
Similarities Between Matching and Statistical Procedures......Page 333
Multivariate Statistical Models......Page 334
Regression-Discontinuity Designs......Page 340
Generic Controls......Page 342
SOME CAUTIONS IN USING CONSTRUCTED CONTROLS......Page 343
Ex Ante Versus Ex Post Designs......Page 345
An Illustrative Ex Post Quasi-Experiment......Page 346
Using Existing Data Sources......Page 347
Using Cross-Sectional Sample Surveys in Ex Post Quasi-Experiments......Page 348
Limitations on the Use of Cross-Sectional Surveys......Page 350
SUMMARY......Page 351
KEY CONCEPTS FOR CHAPTER 10......Page 353
CHAPTER 10 ASSESSMENT OF FULL-COVERAGE PROGRAMS......Page 354
NONUNIFORM FULL-COVERAGE PROGRAMS......Page 355
REFLEXIVE CONTROLS......Page 358
Simple Pre-Post Studies......Page 359
Complex Repeated Measures Reflexive Designs......Page 360
Time-Series Evaluations......Page 363
SHADOW CONTROLS......Page 367
Connoisseurial Assessments......Page 369
Program Administrator Judgments......Page 371
Participant Judgments......Page 372
SUMMARY......Page 374
KEY CONCEPTS FOR CHAPTER 11......Page 375
CHAPTER 11 MEASURING EFFICIENCY......Page 376
KEY CONCEPTS IN EFFICIENCY ANALYSIS......Page 378
Timing of Efficiency Analyses......Page 380
The Concepts of Cost-Benefit and Cost-Effectiveness Analyses......Page 382
Accounting Perspectives......Page 385
Measuring Costs and Benefits......Page 391
Monetizing Outcomes......Page 392
Secondary Effects (Externalities)......Page 394
Distributional Considerations......Page 395
Discounting......Page 396
Ethical Issues in Setting Values......Page 397
Comparing Costs to Benefits......Page 398
When to Do Ex Post Cost-Benefit Analysis......Page 399
COST-EFFECTIVENESS ANALYSIS......Page 401
SUMMARY......Page 405
KEY CONCEPTS FOR CHAPTER 12......Page 407
CHAPTER 12 THE SOCIAL CONTEXT OF EVALUATION......Page 408
THE PURPOSEFULNESS OF EVALUATION ACTIVITIES......Page 409
The Range of Stakeholders......Page 411
Consequences of Multiple Stakeholders......Page 412
Disseminating Evaluation Results......Page 414
Evaluation as a Political Process......Page 417
Political Time and Evaluation Time......Page 418
Policy Relevance and Policy Space......Page 422
Basic Science Models Versus Policy-Oriented Models......Page 425
The Missing Engineering Tradition......Page 426
Evaluating Evaluations......Page 427
THE PROFESSION OF EVALUATION......Page 428
Intellectual Diversity and Its Consequences......Page 429
The Education of Evaluators......Page 430
Consequences of Diversity in Origins......Page 431
Orientations Toward Primary Stakeholders......Page 432
The Qualitative-Quantitative Division......Page 433
Diversity in Working Arrangements......Page 434
Inside Versus Outside Evaluations......Page 435
EVALUATION STANDARDS, GUIDELINES, AND ETHICS......Page 436
The Leadership Role of Evaluation "Elite" Organizations......Page 437
UTILIZATION OF EVALUATION RESULTS......Page 442
Do Evaluations Have Direct Utility?......Page 443
Conceptual Use of Evaluations......Page 444
Variables Affecting Utilization......Page 445
EPILOGUE......Page 447
SUMMARY......Page 450
GLOSSARY......Page 452
REFERENCES......Page 462
AUTHOR INDEX......Page 488
SUBJECT INDEX......Page 494
ABOUT THE AUTHORS......Page 510