Impact evaluation in practice

Cover
Half Title
Title Page
Copyright Page
Contents
Preface
Acknowledgments
About the Authors
Abbreviations
PART ONE. INTRODUCTION TO IMPACT EVALUATION
	Chapter 1. Why Evaluate?
		Evidence-Based Policy Making
		What Is Impact Evaluation?
		Prospective versus Retrospective Impact Evaluation
		Efficacy Studies and Effectiveness Studies
		Complementary Approaches
		Ethical Considerations Regarding Impact Evaluation
		Impact Evaluation for Policy Decisions
		Deciding Whether to Carry Out an Impact Evaluation
	Chapter 2. Preparing for an Evaluation
		Initial Steps
		Constructing a Theory of Change
		Developing a Results Chain
		Specifying Evaluation Questions
		Selecting Outcome and Performance Indicators
		Checklist: Getting Data for Your Indicators
PART TWO. HOW TO EVALUATE
	Chapter 3. Causal Inference and Counterfactuals
		Causal Inference
		The Counterfactual
		Two Counterfeit Estimates of the Counterfactual
	Chapter 4. Randomized Assignment
		Evaluating Programs Based on the Rules of Assignment
		Randomized Assignment of Treatment
		Checklist: Randomized Assignment
	Chapter 5. Instrumental Variables
		Evaluating Programs When Not Everyone Complies with Their Assignment
		Types of Impact Estimates
		Imperfect Compliance
		Randomized Promotion as an Instrumental Variable
		Checklist: Randomized Promotion as an Instrumental Variable
	Chapter 6. Regression Discontinuity Design
		Evaluating Programs That Use an Eligibility Index
		Fuzzy Regression Discontinuity Design
		Checking the Validity of the Regression Discontinuity Design
		Limitations and Interpretation of the Regression Discontinuity Design Method
		Checklist: Regression Discontinuity Design
	Chapter 7. Difference-in-Differences
		Evaluating a Program When the Rule of Assignment Is Less Clear
		The Difference-in-Differences Method
		How Is the Difference-in-Differences Method Helpful?
		The “Equal Trends” Assumption in Difference-in-Differences
		Limitations of the Difference-in-Differences Method
		Checklist: Difference-in-Differences
	Chapter 8. Matching
		Constructing an Artificial Comparison Group
		Propensity Score Matching
		Combining Matching with Other Methods
		Limitations of the Matching Method
		Checklist: Matching
	Chapter 9. Addressing Methodological Challenges
		Heterogeneous Treatment Effects
		Unintended Behavioral Effects
		Imperfect Compliance
		Spillovers
		Attrition
		Timing and Persistence of Effects
	Chapter 10. Evaluating Multifaceted Programs
		Evaluating Programs That Combine Several Treatment Options
		Evaluating Programs with Varying Treatment Levels
		Evaluating Multiple Interventions
PART THREE. HOW TO IMPLEMENT AN IMPACT EVALUATION
	Chapter 11. Choosing an Impact Evaluation Method
		Determining Which Method to Use for a Given Program
		How a Program’s Rules of Operation Can Help Choose an Impact Evaluation Method
		A Comparison of Impact Evaluation Methods
		Finding the Smallest Feasible Unit of Intervention
	Chapter 12. Managing an Impact Evaluation
		Managing an Evaluation’s Team, Time, and Budget
		Roles and Responsibilities of the Research and Policy Teams
		Establishing Collaboration
		How to Time the Evaluation
		How to Budget for an Evaluation
	Chapter 13. The Ethics and Science of Impact Evaluation
		Managing Ethical and Credible Evaluations
		The Ethics of Running Impact Evaluations
		Ensuring Reliable and Credible Evaluations through Open Science
		Checklist: An Ethical and Credible Impact Evaluation
	Chapter 14. Disseminating Results and Achieving Policy Impact
		A Solid Evidence Base for Policy
		Tailoring a Communication Strategy to Different Audiences
		Disseminating Results
PART FOUR. HOW TO GET DATA FOR AN IMPACT EVALUATION
	Chapter 15. Choosing a Sample
		Sampling and Power Calculations
		Drawing a Sample
		Deciding on the Size of a Sample for Impact Evaluation: Power Calculations
	Chapter 16. Finding Adequate Sources of Data
		Kinds of Data That Are Needed
		Using Existing Quantitative Data
		Collecting New Survey Data
	Chapter 17. Conclusion
		Impact Evaluations: Worthwhile but Complex Exercises
		Checklist: Core Elements of a Well-Designed Impact Evaluation
		Checklist: Tips to Mitigate Common Risks in Conducting an Impact Evaluation
Glossary
Boxes
	1.1 How a Successful Evaluation Can Promote the Political Sustainability of a Development Program: Mexico’s Conditional Cash Transfer Program
	1.2 The Policy Impact of an Innovative Preschool Model: Preschool and Early Childhood Development in Mozambique
	1.3 Testing for the Generalizability of Results: A Multisite Evaluation of the “Graduation” Approach to Alleviate Extreme Poverty
	1.4 Simulating Possible Project Effects through Structural Modeling: Building a Model to Test Alternative Designs Using Progresa Data in Mexico
	1.5 A Mixed Method Evaluation in Action: Combining a Randomized Controlled Trial with an Ethnographic Study in India
	1.6 Informing National Scale-Up through a Process Evaluation in Tanzania
	1.7 Evaluating Cost-Effectiveness: Comparing Evaluations of Programs That Affect Learning in Primary Schools
	1.8 Evaluating Innovative Programs: The Behavioural Insights Team in the United Kingdom
	1.9 Evaluating Program Design Alternatives: Malnourishment and Cognitive Development in Colombia
	1.10 The Impact Evaluation Cluster Approach: Strategically Building Evidence to Fill Knowledge Gaps
	2.1 Articulating a Theory of Change: From Cement Floors to Happiness in Mexico
	2.2 Mechanism Experiments
	2.3 A High School Mathematics Reform: Formulating a Results Chains and Evaluation Question
	3.1 The Counterfactual Problem: “Miss Unique” and the Cash Transfer Program
	4.1 Randomized Assignment as a Valuable Operational Tool
	4.2 Randomized Assignment as a Program Allocation Rule: Conditional Cash Transfers and Education in Mexico
	4.3 Randomized Assignment of Grants to Improve Employment Prospects for Youth in Northern Uganda
	4.4 Randomized Assignment of Water and Sanitation Interventions in Rural Bolivia
	4.5 Randomized Assignment of Spring Water Protection to Improve Health in Kenya
	4.6 Randomized Assignment of Information about HIV Risks to Curb Teen Pregnancy in Kenya
	5.1 Using Instrumental Variables to Evaluate the Impact of Sesame Street on School Readiness
	5.2 Using Instrumental Variables to Deal with Noncompliance in a School Voucher Program in Colombia
	5.3 Randomized Promotion of Education Infrastructure Investments in Bolivia
	6.1 Using Regression Discontinuity Design to Evaluate the Impact of Reducing School Fees on School Enrollment Rates in Colombia
	6.2 Social Safety Nets Based on a Poverty Index in Jamaica
	6.3 The Effect on School Performance of Grouping Students by Test Scores in Kenya
	7.1 Using Difference-in-Differences to Understand the Impact of Electoral Incentives on School Dropout Rates in Brazil
	7.2 Using Difference-in-Differences to Study the Effects of Police Deployment on Crime in Argentina
	7.3 Testing the Assumption of Equal Trends: Water Privatization and Infant Mortality in Argentina
	7.4 Testing the Assumption of Equal Trends: School Construction in Indonesia
	8.1 Matched Difference-in-Differences: Rural Roads and Local Market Development in Vietnam
	8.2 Matched Difference-in-Differences: Cement Floors, Child Health, and Maternal Happiness in Mexico
	8.3 The Synthetic Control Method: The Economic Effects of a Terrorist Conflict in Spain
	9.1 Folk Tales of Impact Evaluation: The Hawthorne Effect and the John Henry Effect
	9.2 Negative Spillovers Due to General Equilibrium Effects: Job Placement Assistance and Labor Market Outcomes in France
	9.3 Working with Spillovers: Deworming, Externalities, and Education in Kenya
	9.4 Evaluating Spillover Effects: Conditional Cash Transfers and Spillovers in Mexico
	9.5 Attrition in Studies with Long-Term Follow-Up: Early Childhood Development and Migration in Jamaica
	9.6 Evaluating Long-Term Effects: Subsidies and Adoption of Insecticide-Treated Bed Nets in Kenya
	10.1 Testing Program Intensity for Improving Adherence to Antiretroviral Treatment
	10.2 Testing Program Alternatives for Monitoring Corruption in Indonesia
	11.1 Cash Transfer Programs and the Minimum Level of Intervention
	12.1 Guiding Principles for Engagement between the Policy and Evaluation Teams
	12.2 General Outline of an Impact Evaluation Plan
	12.3 Examples of Research–Policy Team Models
	13.1 Trial Registries for the Social Sciences
	14.1 The Policy Impact of an Innovative Preschool Model in Mozambique
	14.2 Outreach and Dissemination Tools
	14.3 Disseminating Impact Evaluations Effectively
	14.4 Disseminating Impact Evaluations Online
	14.5 Impact Evaluation Blogs
	15.1 Random Sampling Is Not Sufficient for Impact Evaluation
	16.1 Constructing a Data Set in the Evaluation of Argentina’s Plan Nacer
	16.2 Using Census Data to Reevaluate the PRAF in Honduras
	16.3 Designing and Formatting Questionnaires
	16.4 Some Pros and Cons of Electronic Data Collection
	16.5 Data Collection for the Evaluation of the Atención a Crisis Pilots in Nicaragua
	16.6 Guidelines for Data Documentation and Storage
Figures
	2.1 The Elements of a Results Chain
	B2.2.1 Identifying a Mechanism Experiment from a Longer Results Chain
	B2.3.1 A Results Chain for the High School Mathematics Curriculum Reform
	2.2 The HISP Results Chain
	3.1 The Perfect Clone
	3.2 A Valid Comparison Group
	3.3 Before-and-After Estimates of a Microfinance Program
	4.1 Characteristics of Groups under Randomized Assignment of Treatment
	4.2 Random Sampling and Randomized Assignment of Treatment
	4.3 Steps in Randomized Assignment to Treatment
	4.4 Using a Spreadsheet to Randomize Assignment to Treatment
	4.5 Estimating Impact under Randomized Assignment
	5.1 Randomized Assignment with Imperfect Compliance
	5.2 Estimating the Local Average Treatment Effect under Randomized Assignment with Imperfect Compliance
	5.3 Randomized Promotion
	5.4 Estimating the Local Average Treatment Effect under Randomized Promotion
	6.1 Rice Yield, Smaller Farms versus Larger Farms (Baseline)
	6.2 Rice Yield, Smaller Farms versus Larger Farms (Follow-Up)
	6.3 Compliance with Assignment
	6.4 Manipulation of the Eligibility Index
	6.5 HISP: Density of Households, by Baseline Poverty Index
	6.6 Participation in HISP, by Baseline Poverty Index
	6.7 Poverty Index and Health Expenditures, HISP, Two Years Later
	7.1 The Difference-in-Differences Method
	7.2 Difference-in-Differences When Outcome Trends Differ
	8.1 Exact Matching on Four Characteristics
	8.2 Propensity Score Matching and Common Support
	8.3 Matching for HISP: Common Support
	9.1 A Classic Example of Spillovers: Positive Externalities from Deworming School Children
	10.1 Steps in Randomized Assignment of Two Levels of Treatment
	10.2 Steps in Randomized Assignment of Two Interventions
	10.3 Crossover Design for a Program with Two Interventions
	15.1 Using a Sample to Infer Average Characteristics of the Population of Interest
	15.2 A Valid Sampling Frame Covers the Entire Population of Interest
	B15.1.1 Random Sampling among Noncomparable Groups of Participants and Nonparticipants
	B15.1.2 Randomized Assignment of Program Benefits between a Treatment Group and a Comparison Group
	15.3 A Large Sample Is More Likely to Resemble the Population of Interest
Tables
	3.1 Evaluating HISP: Before-and-After Comparison
	3.2 Evaluating HISP: Before-and-After with Regression Analysis
	3.3 Evaluating HISP: Enrolled-Nonenrolled Comparison of Means
	3.4 Evaluating HISP: Enrolled-Nonenrolled Regression Analysis
	4.1 Evaluating HISP: Balance between Treatment and Comparison Villages at Baseline
	4.2 Evaluating HISP: Randomized Assignment with Comparison of Means
	4.3 Evaluating HISP: Randomized Assignment with Regression Analysis
	5.1 Evaluating HISP: Randomized Promotion Comparison of Means
	5.2 Evaluating HISP: Randomized Promotion with Regression Analysis
	6.1 Evaluating HISP: Regression Discontinuity Design with Regression Analysis
	7.1 Calculating the Difference-in-Differences (DD) Method
	7.2 Evaluating HISP: Difference-in-Differences Comparison of Means
	7.3 Evaluating HISP: Difference-in-Differences with Regression Analysis
	8.1 Estimating the Propensity Score Based on Baseline Observed Characteristics
	8.2 Evaluating HISP: Matching on Baseline Characteristics and Comparison of Means
	8.3 Evaluating HISP: Matching on Baseline Characteristics and Regression Analysis
	8.4 Evaluating HISP: Difference-in-Differences Combined with Matching on Baseline Characteristics
	B10.1.1 Summary of Program Design
	11.1 Relationship between a Program’s Operational Rules and Impact Evaluation Methods
	11.2 Comparing Impact Evaluation Methods
	12.1 Cost of Impact Evaluations of a Selection of World Bank–Supported Projects
	12.2 Disaggregated Costs of a Selection of World Bank–Supported Impact Evaluations
	12.3 Sample Budget for an Impact Evaluation
	13.1 Ensuring Reliable and Credible Information for Policy through Open Science
	14.1 Engaging Key Constituencies for Policy Impact: Why, When, and How
	15.1 Examples of Clusters
	15.2 Evaluating HISP+: Sample Size Required to Detect Various Minimum Detectable Effects, Power = 0.9
	15.3 Evaluating HISP+: Sample Size Required to Detect Various Minimum Detectable Effects, Power = 0.8
	15.4 Evaluating HISP+: Sample Size Required to Detect Various Minimum Desired Effects (Increase in Hospitalization Rate)
	15.5 Evaluating HISP+: Sample Size Required to Detect Various Minimum Detectable Effects (Decrease in Household Health Expenditures)
	15.6 Evaluating HISP+: Sample Size Required to Detect a US$2 Minimum Impact for Various Numbers of Clusters