Ten Steps to a Results-Based Monitoring and Evaluation System: A Handbook for Development Practitioners

Contents......Page 7
Preface......Page 13
About the Authors......Page 16
Building a Results-Based Monitoring and Evaluation System......Page 17
New Challenges in Public Sector Management......Page 18
International and External Initiatives and Forces for Change......Page 19
National Poverty Reduction Strategy Approach......Page 24
Internal Initiatives and Forces for Change......Page 26
i.iv The Power of Measuring Results......Page 27
Monitoring and Evaluation: What Is It All About?......Page 28
Key Features of Traditional Implementation-Focused and Results-Based M&E Systems......Page 31
i.v Key Features of Implementation Monitoring versus Results Monitoring......Page 33
Political and Technical Challenges to Building a Results-Based M&E System......Page 36
Introducing the 10-Step Model for Building a Results-Based M&E System......Page 39
Where to Begin: Whole-of-Government, Enclave, or Mixed Approach......Page 40
M&E Experience in Developed and OECD Countries......Page 43
Special M&E Challenges Facing Developing Countries......Page 48
M&E Experience in Developing Countries......Page 51
1.1 Conducting a Readiness Assessment......Page 55
Why Do a Readiness Assessment?......Page 56
The Readiness Assessment: Eight Key Questions......Page 59
Readiness Assessments in Developing Countries: Bangladesh, Egypt, and Romania......Page 64
Lessons Learned......Page 65
2.1 Agreeing on Outcomes to Monitor and Evaluate......Page 72
Issues to Consider in Choosing Outcomes to Monitor and Evaluate......Page 73
The Importance of Building a Participatory and Consultative Process involving Main Stakeholders......Page 74
The Overall Process of Setting and Agreeing upon Outcomes......Page 75
Examples and Possible Approaches......Page 77
3.1 Selecting Key Indicators to Monitor Outcomes......Page 81
Translating Outcomes into Outcome Indicators......Page 82
3.2 Developing a Set of Outcome Indicators for a Policy Area......Page 84
The Use of Proxy Indicators......Page 86
The Pros and Cons of Using Predesigned Indicators......Page 88
Constructing Indicators......Page 90
Setting Indicators: Experience in Developing Countries......Page 91
4.1 Baseline Data on Indicators—Where Are We Today?......Page 96
4.2 Developing Baseline Data for One Policy Area......Page 97
4.1 Building Baseline Information......Page 98
Identifying Data Sources for Indicators......Page 99
Designing and Comparing Data Collection Methods......Page 100
The Importance of Conducting Pilots......Page 102
4.2 Lebanon: Joining the IMF Data System......Page 105
5.1 Planning for Improvement—Selecting Results Targets......Page 106
5.2 Identifying Desired Level of Results Requires Selecting Performance Targets......Page 107
Examples of Targets Related to Development Issues......Page 109
5.1 Examples of Development Targets......Page 110
6.1 Monitoring for Results......Page 112
Key Types and Levels of Monitoring......Page 114
6.1 Results Monitoring in Mexico......Page 117
6.5 Links between Implementation Monitoring and Results Monitoring......Page 119
Achieving Results through Partnership......Page 121
6.7 Achieving Results through Partnership......Page 122
The Data Quality Triangle: Reliability, Validity, and Timeliness......Page 124
6.13 Analyzing Results Data......Page 127
Pretesting Data Collection Instruments and Procedures......Page 128
7.1 The Role of Evaluations......Page 129
Uses of Evaluation......Page 131
7.2 Using Evaluation to Explain Performance Divergence......Page 134
7.4 Seven Types of Evaluations......Page 137
7.5 Characteristics of Quality Evaluations......Page 142
7.6 Examples of Evaluation......Page 144
8.1 Reporting Findings......Page 145
Know and Target the Audience......Page 146
Presentation of Performance Data in Clear and Understandable Form......Page 148
What Happens If the M&E System Produces Bad Performance News?......Page 152
9.1 Using Findings......Page 154
Additional Benefits of Using Findings: Feedback, Knowledge, and Learning......Page 156
9.6 Incentives for Learning, Knowledge Building, and Greater Use of Performance Findings......Page 162
10.1 Sustaining the M&E System within the Organization......Page 167
Six Critical Components of Sustaining Results-Based M&E Systems......Page 168
10.1 Citizen’s Charter in the United Kingdom......Page 171
Last Reminders......Page 176
Why Results-Based M&E?......Page 178
How to Create Results-Based M&E Systems......Page 181
Summing Up......Page 186
Annexes:......Page 189
Annex I: Assessing Performance-Based Monitoring and Evaluation Capacity: An Assessment Survey for Countries, Development Institutions, and Their Partners......Page 190
Annex II: Readiness Assessment: Toward Results-Based Monitoring and Evaluation in Egypt......Page 194
Annex III: Millennium Development Goals (MDGs): List of Goals and Targets......Page 216
Annex IV: National Evaluation Policy for Sri Lanka: Sri Lanka Evaluation Association (SLEva) jointly with the Ministry of Policy Development and Implementation......Page 220
Annex V: Andhra Pradesh (India) Performance Accountability Act 2003: (Draft Act) (APPAC Act of 2003)......Page 227
Annex VI: Glossary: OECD Glossary of Key Terms in Evaluation and Results-Based Management (2002)......Page 239
Notes......Page 246
References......Page 247
Useful Web Sites......Page 251
Additional Reading......Page 252
Index......Page 255
i.i Millennium Development Goals......Page 20
i.ii Example of Millennium Development Goal, Targets, and Indicators......Page 21
i.iii Transparency International......Page 22
i.vi Australia’s Whole-of-Government Model......Page 45
i.vii France: Lagging Behind but Now Speeding Ahead in Governmental Reform......Page 46
i.viii Republic of Korea: Well on the Road to M&E......Page 47
i.ix Malaysia: Outcome-Based Budgeting, Nation Building, and Global Competitiveness......Page 52
i.x Uganda and Poverty Reduction—Impetus toward M&E......Page 53
1.1 The Case of Bangladesh—Building from the Bottom Up......Page 66
1.2 The Case of Egypt—Slow, Systematic Moves toward M&E......Page 67
1.3 The Case of Romania—Some Opportunities to Move toward M&E......Page 68
3.3 Checklist for Assessing Proposed Indicators......Page 87
3.2 The Africa Region’s Core Welfare Indicators......Page 92
3.3 Sri Lanka’s National Evaluation Policy......Page 93
3.4 Albania’s Three-Year Action Plan......Page 94
3.6 Outcome: Increased Participation of Farmers in Local Markets......Page 95
4.1 Albania’s Strategy for Strengthening Data Collection Capacity......Page 104
6.2 Results Monitoring in Brazil......Page 118
7.1 Evaluation Provides Information on Strategy, Operations, and Learning......Page 133
9.1 Ten Uses of Results Findings......Page 155
9.2 Using Performance Data to Track and Reduce Crime in New York City......Page 157
9.3 U.S. Department of Labor—An Organization with a Mature, Functioning Results-Based M&E System......Page 158
9.4 Signs of Improving Conditions for Evaluation-Based Learning in German Aid Agencies......Page 160
9.5 Obstacles to Learning......Page 161
9.7 Active and Passive Approaches to Using Results Information......Page 163
9.8 Canadian Government Performance Reports to Parliament......Page 165
10.2 U.S. Government Performance and Results Act of 1993......Page 172
10.4 Checklist for Staff Disincentives That Hinder Learning-Oriented, Participatory M&E......Page 174
10.5 An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity......Page 177
i.i Complementary Roles of Results-Based Monitoring and Evaluation......Page 30
4.2 Comparison of Major Data Collection Methods......Page 103
8.1 Outcomes Reporting Format: Actual Outcomes versus Targets......Page 149
8.2 Sample Table for Reporting Descriptive Data: Gender Differences in Voting......Page 151
10.1 Evaluation Capacity Development and Institutionalization—Key Issues Addressed in Colombia, China, and Indonesia......Page 173
i.i Illustrative Logic Model for One National Development Goal......Page 34
i.ii Ten Steps to Designing, Building, and Sustaining a Results-Based Monitoring and Evaluation System......Page 41
2.2 Developing Outcome Statements......Page 76
2.3 Outcome Statements Derived from Identified Problems or Issues......Page 78
2.4 How NOT to Construct Outcome Statements......Page 79
2.5 Developing Outcomes for One Policy Area......Page 80
4.3 Data Collection Methods......Page 101
5.3 Developing Targets for One Policy Area......Page 111
6.2 Sample Gant Chart......Page 113
6.3 Results-Based Monitoring......Page 115
6.4 Examples of Results Monitoring......Page 116
6.6 Linking Implementation Monitoring to Results Monitoring......Page 120
6.8 Every Monitoring System Needs Ownership, Management, Maintenance, and Credibility......Page 123
6.10 The Data Quality Triangle: Reliability......Page 125
6.12 The Data Quality Triangle: Timeliness......Page 126
7.3 Using Evaluation to Determine the Impacts of Design and Implementation on Outcome......Page 135
8.2 Principles of Graphic Excellence and Sample Charts for Displaying Information......Page 153