Running Randomized Evaluations: A Practical Guide - Softcover

Glennerster, Rachel; Takavarasha, Kudzai

Softcover

ISBN 10: 0691159270 ISBN 13: 9780691159270

Publisher: Princeton University Press, 2013

Synopsis

A comprehensive guide to running randomized impact evaluations of social programs in developing countries

This book provides a comprehensive yet accessible guide to running randomized impact evaluations of social programs. Drawing on the experience of researchers at the Abdul Latif Jameel Poverty Action Lab, which has run hundreds of such evaluations in dozens of countries throughout the world, it offers practical insights on how to use this powerful technique, especially in resource-poor environments.

This step-by-step guide explains why and when randomized evaluations are useful, in what situations they should be used, and how to prioritize different evaluation opportunities. It shows how to design and analyze studies that answer important questions while respecting the constraints of those working on and benefiting from the program being evaluated. The book gives concrete tips on issues such as improving the quality of a study despite tight budget constraints, and demonstrates how the results of randomized impact evaluations can inform policy.

With its self-contained modules, this one-of-a-kind guide is easy to navigate. It also includes invaluable references and a checklist of the common pitfalls to avoid.

• Provides the most up-to-date guide to running randomized evaluations of social programs, especially in developing countries
• Offers practical tips on how to complete high-quality studies in even the most challenging environments
• Self-contained modules allow for easy reference and flexible teaching and learning
• Comprehensive yet nontechnical

About this title

About the Author

Rachel Glennerster is executive director of the Abdul Latif Jameel Poverty Action Lab (J-PAL) at the Massachusetts Institute of Technology. Kudzai Takavarasha holds degrees in chemical engineering and economics from the Massachusetts Institute of Technology, and was a policy manager at J-PAL from 2004 to 2012.

From the Back Cover

"The popularity of randomized evaluations among researchers and policymakers is growing and holds great promise for a world where decision making will be based increasingly on rigorous evidence and creative thinking. However, conducting a randomized evaluation can be daunting. There are many steps, and decisions made early on can have unforeseen implications for the life of the project. This book, based on more than a decade of personal experience by a foremost practitioner and a wealth of knowledge gathered over the years by researchers at J-PAL, provides both comfort and guidance to anyone seeking to engage in this process."--Esther Duflo, codirector of J-PAL and coauthor of Poor Economics

"Randomized impact evaluations, pioneered by Rachel Glennerster and her colleagues, have emerged as a powerful new tool in the fight against poverty. This book gives us a critical guide for how to actually do them. It is required reading for students, professionals, and researchers committed to using evidence to guide policy."--Dean Karlan, coauthor of More Than Good Intentions

"I have been searching for a book to train my students, research assistants, and field staff to design and implement social policy experiments. My search is over. Running Randomized Evaluations provides practical guidance covering all stages of an experiment--wisdom that previously could be acquired only by working directly for an evaluation expert."--Jeffrey B. Liebman, John F. Kennedy School of Government, Harvard University

"Impact evaluation is a key resource for improving development effectiveness, yet until now it was inaccessible to people--policymakers, NGO staff, and development professionals of all kinds. Oftentimes, we get stuck in the theoretical discourse and philosophical debate and miss the chance to talk about the practical toolkit needed to actually do impact evaluation. Running Randomized Evaluations will significantly contribute to accelerate advancements in the field by making the tools of the trade accessible and user-friendly. This is a must-read guide for policymakers, practitioners, and donors alike."--Jodi Nelson, Bill and Melinda Gates Foundation

Excerpt. © Reprinted by permission. All rights reserved.

Running Randomized Evaluations

A PRACTICAL GUIDE

By RACHEL GLENNERSTER, KUDZAI TAKAVARASHA

PRINCETON UNIVERSITY PRESS

Copyright © 2013 Princeton University Press
All rights reserved.

ISBN: 978-0-691-15927-0

Contents

Preface....................................................................
vii
Abbreviations and Acronyms.................................................
ix
1 The Experimental Approach................................................
1
2 Why Randomize?...........................................................
24
3 Asking the Right Questions...............................................
66
4 Randomizing..............................................................
98
5 Outcomes and Instruments.................................................
180
6 Statistical Power........................................................
241
7 Threats..................................................................
298
8 Analysis.................................................................
324
9 Drawing Policy Lessons...................................................
386
Appendix...................................................................
421
Glossary...................................................................
443
Index......................................................................
453

CHAPTER 1

The Experimental Approach

This chapter provides an example of how a randomized evaluationcan lead to large-scale change and provides a road map for an evaluationand for the rest of the book. The modules in this chapter are asfollows:

MODULE 1.1: The Power of Randomized Evaluations

MODULE 1.2: A Randomized Evaluation from Start to Finish

MODULE 1.1 The Power of Randomized Evaluations

This module provides an example of how a small nongovernmental organization,by subjecting its program to rigorous evaluation, can generateevidence that can change the lives of millions of people.

In 1994 I went with Michael Kremer to visit the family he had livedwith for a year in rural Kenya. We met up with many of Michael's oldfriends, including Paul Lipeyah, who told us of the work he was doingwith International Child Support (ICS) Africa, a nongovernmentalorganization (NGO) helping government schools in Busia, a neighboringdistrict in Kenya's Western Province. Paul asked us what advicewe might offer for improving the effectiveness of ICS programs. CouldMichael help evaluate what they were doing? Michael suggested randomizedevaluation: if ICS wanted to understand the impact of theirprograms, they could randomly choose the schools in which theyworked and the order in which they phased in new programs.

Over the following years, ICS randomly evaluated many approachesto improving education outcomes, including providing additional inputs(classrooms, textbooks, teachers); reducing the cost of going toschool (providing free school uniforms and school meals); and offeringperformance incentives (merit scholarships for girls, bonuses forteachers who attended school regularly). Sometimes the programshad the expected impact, and sometimes they did not. But ICS, inpartnership with a growing number of researchers, kept innovatingand testing in the areas of education, agriculture, women's empowerment,clean water, and health. Their knowledge of how to improvelives and how to evaluate programs kept growing.

In 1997, ICS (in line with World Health Organization recommendations)phased in a program of treating children en massefor intestinal worms (such as hookworm and schistosomiasis). Theresults were astonishing. Deworming reduced the absenteeism ofchildren in local schools by 25 percent, making it the most cost-effectiveprogram for increasing schooling that ICS had tried. Long-termresults showed that women who had been dewormed as girlsreceived more education and were more likely to grow cash crops,whereas men who had been dewormed worked 3.5 hours longer perweek and were more likely to hold manufacturing jobs and earnhigher wages.

On the strength of the evidence, in 2009 Prime Minister RailaOdinga announced a program to deworm 3 million of Kenya's mostat-risk children. In 2012 the program was expanded to include preschoolchildren, in part on the basis of further evidence of cogitativegains for young children from deworming. In 2013, programs to deworm40 million children are being implemented in Kenya andaround the world. ICS could never have hoped to reach so many childrenwith their own programs, and yet through the influence of theirevaluation they have helped millions of children in Kenya—andaround the world.

Since 1994 we have learned a lot about which programs work andwhich do not and also about how to run randomized evaluations.Until that point, most randomized evaluations of social programswere performed in rich countries in partnership with governmentsand at very high cost. But the experience of partnerships betweenresearchers and organizations such as ICS showed that it was possibleto conduct high-quality randomized evaluations with small organizationson limited budgets and in very poor settings. Although manygovernment programs are being evaluated with randomized evaluations,NGOs have also proved to be flexible and innovative partners inthis effort to learn. The challenge of working with new partners onnew questions and on modest budgets has spurred innovation in randomizedevaluation methodology.

We have learned how to introduce randomization into programsin creative ways, account for and measure spillovers, reliably measuredifficult-to-measure outcomes like corruption and empowerment,get the maximum statistical power from a very limited budget, minimizeattrition, and design evaluations that answer fundamental questionsabout why humans behave the way they do and how to motivatechanges in behavior.

In this book we have gathered many of the practical innovationsfrom this large body of work. Our goal is to enable more people andorganizations to undertake and commission high-quality randomizedimpact evaluations and thus to build a wider and deeper evidencebase on how best to combat poverty. Our hope is that we will see thegeneration and application of rigorous evidence grow even faster thanin the past two decades. By innovating and testing and by feeding theevidence back into even more innovation, evaluators and practitionerscan improve the effectiveness of policies and programs and makea real difference in people's lives.

MODULE 1.2 A Randomized Evaluation from Start to Finish

In this module we provide an overview of the steps in planning and runningan evaluation and indicate where these topics are covered in therest of this book. We tell the story of a randomized evaluation of an educationprogram in India that I experienced firsthand from inceptionthrough implementation and analysis to scale-up.

Starting right: Choosing the right question to test

Ten years after my trip to Kenya described in Module 1.1 , I was workingwith a group of researchers at the Massachusetts Institute of Technology(MIT) planning our next evaluation. We had recently startedwhat is now the Abdul Latif Jameel Poverty Action Lab (J-PAL) withthe objective of promoting the use of randomized evaluations andhelping ensure that the results are used to influence policy. Althoughrandomized evaluations (and most of the lessons in this book) arevaluable across many disciplines and many regions of the world, ourmain expertise was in development economics. In prioritizing our work,therefore, we wanted to start by understanding the areas in which rigorousevaluation could be most valuable in informing the debateabout poverty in developing countries. Which innovations showedpromise but were untested? Which programs were popular with governmentsand NGOs but had little rigorous evidence to support them?

Community accountability programs were a priority that repeatedlyemerged in the conversations we had with organizations workingin developing countries as well as in our review of the literature. Theenthusiasm for this approach was well articulated in World DevelopmentReport 2004: Making Services Work for Poor People. The reportdocumented the low quality of services and the lack of accountability,including the chronically high absenteeism of service providers. Itargued that community accountability is one of the best ways toimprove failing services. The poor who suffered the brunt of the failureswere not just more motivated to get services working than werebureaucrats; they were also better positioned because they were rightthere, at the point of delivery, to monitor the providers. If they wereempowered to apply their motivation and monitoring advantages,they would hold providers accountable and services would improve.

In practice, this empowerment took the form of establishing communityoversight bodies for schools and clinics and providing communitieswith information about their rights and the services they shouldexpect. International agencies, NGOs, and governments were all lookingto integrate the approach into their work.

There were reasons to think that community accountability wouldwork in practice. Advocates pointed to correlations between activeparticipation of citizens in school and clinic oversight and high-qualityservices, and cases in which increases in participation wereassociated with improved services. A popular example was documentedin a study from Uganda, in which the government had starteddisbursing grants directly to schools. A survey found that only 25 percentof these grants were reported as reaching the schools. In response,the government started informing communities of how much moneyhad been allocated to each school, and a few years later, 82 percent ofthe grants were reported to be reaching the schools.

It was unclear, however, whether the correlation between communityinvolvement and high-quality service outcomes meant that communityinvolvement caused these high-quality outcomes. Communitieswith high levels of citizeninvolvement tend to be differentfrom those with low levelsof involvement in a number ofways. For example, the townof Brookline, Massachusetts,where I live, has very goodpublic schools and an unusualform of governance in whichcitizens take responsibilities typically given to full-time town employees.But Brookline also has an unusual concentration of people withhigh levels of education. People move from miles away and pay hightaxes so that their children can attend Brookline's public schools. It ishard to know to what extent the school outcomes are due to the citizenoversight and to what extent they are due to the emphasis on educationamong local families. More important, it is not clear whetheranother town encouraged (or mandated) to take up the Brooklinemodel of citizen involvement would achieve the same outcomes.

What about the Uganda example, in which services improved whenthe information given to local people was increased? Even there it wasunclear how big a role empowering communities with informationplayed in the observed changes. Some people believed that the accuracywith which money transfers to schools were recorded had been lowduring the first year and improved over time. In addition, informationon how few of the grant funds made it to the schools was also reportedto the Ministry of Education and to donors, and that caused quite a stir.Was it providing information to the ministry and the donors or providingit to the community that caused the change? It's hard to tell.

Because the approach of empowering communities to hold serviceproviders to account was popular and there was little rigorous evidenceof its impact, we decided to prioritize this as one of the questionswe wanted to test using a randomized evaluation.

Finding a specific context in which to test the question

Among those keen to both develop and test a community accountabilityprogram was Pratham. Pratham is the largest organization,apart from the government, working on education in India. Was Indiathe right context? Was Pratham the right partner?

The education sector in India was plagued by poor public services.The absence rate among primary school teachers was 25 percent,and only 45 percent of teachers present were in the classroom teaching.Pratham had found in other work that even many children whoattended school regularly could not read or do simple math by grades3 and 4. Services were bad and highly centralized; there was scope,therefore, for community accountability to make a difference. The researchersdecided that India was a relevant context in which to test acommunity accountability program.

Pratham was, in fact, an ideal evaluation partner. They knew a lotabout education in India, and they wanted to test a community mobilizationprogram. Pratham had previously worked with another J-PALteam, including Abhijit Banerjee and Esther Duflo, to evaluate a remedialeducation program that trained local young women and placedthem in local schools as tutors for children who had fallen behind.The evaluation had found the program highly effective in helpingthese children catch up in reading and arithmetic. Pratham wanted toadapt this model to a rural setting and reduce costs by relying on volunteersrather than paid tutors. Successful as their program hadbeen, Pratham also believed that improving the education of childrenin India required improving the quality of education in the governmentschools on which most children relied. They had a vision of citizenscoming together to ensure that India's children had a better futureboth by exerting pressure to improve public education and by takingdirect action to improve learning. Further, Pratham understood whyrandomized evaluations were useful, they were in the early designphase of a new program allowing the joint development of the researchand project design, and they had the ability to bring a successful programto scale. The research team also recruited Stuti Khemani at theWorld Bank to bring to the project the expertise and perspective ofthe World Bank on community accountability programs.

Together, Pratham and the research team decided that Uttar Pradesh(UP) would be a good location for the program and its evaluation. UPis one of India's largest states, with 20 million primary school–agedchildren. But school quality there was very low: the survey we conductedat the start of the evaluation (our baseline) showed that only42 percent of 7- to 14-year-olds could read and understand a simplestory. Because it would not be feasible to introduce community oversightboards with legal backing to some (randomly selected) schoolsand not others, we needed a context in which such an oversight boardexisted but was not very active. UP had legislation mandating thatVECs oversee all public schools in a village, but most did not function.Laws existed that gave communities a number of paths to influencethe quality of their schools: communities could complain totheir members of parliament, local village councils could requestfunding to hire local assistant teachers, and the village councils haddiscretionary funds they could use to improve local services. WithinUP we chose to pilot and evaluate the program in Jaunpur District,which was close to the state average in terms of literacy and was oneof the districts where Pratham was not already working at the time.

Given our chosen context,we refined our general communityaccountability questionto this: is there a way to mobilizecommunities to effectivelyuse existing accountability systemsin UP to improve educationalquality?

The groundwork: How did we arrive at the final three interventions?

Over the period of a year, the researchers and Pratham worked togetherto design the program and the evaluation. Their objective wasa program that represented best practices in the area of communitymobilization to enhance service accountability and was tailored to thelocal environment but was replicable. We also had to select which ofseveral alternative versions of the program to rigorously test againsteach other in the study. Finally, we had to determine how to measurethe impact of the program.

Honing the design of the program

Both Pratham and the researchers wanted to design and test a programthat would be replicable at large scale. In other words, it neededto be relatively inexpensive and not rely on highly trained or educatedstaff. Scalability concerns also limited the resources Pratham wouldput into any single village mobilization effort.

We needed to check whether the theory behind the interventionmade sense in Jaunpur District. Was the level of learning low? Wasthere room to improve education (for example, was the rate of teacherabsenteeism high)? Did communities have mechanisms they coulduse to improve education quality? What were the roles and responsibilitiesof the VECs? If a village wanted an additional assistant teacheror wanted to fire an existing one, how exactly could they do that? Howmuch money did the village council have that could be directed toeducation? Was there a gap in knowledge that a potential interventioncould fill? Did the community know how poor learning levels were?Did everyone know the roles and responsibilities of the VEC? Wasthere relevant information that the village head knew but was notsharing widely (in which case pressing the village head to share thatinformation might be a good strategy)? We spent time in governmentoffices in the state capital of Lucknow finding out exactly what lawswere on the books and talked to village heads, teachers, VEC members,students, community members, and national and state educationactivists.

We found that learning was very poor in Jaunpur, but most peopleoverestimated the level of learning in their community and wereshocked when they realized how little their children knew. Accordingto the law, communities had a number of ways they could press forchange, most of which ran through the VECs; yet there was very littleknowledge of the VECs, what their powers were, how much moneythey had, or even who was on them. Although village heads tended toknow about VECs and could produce a list of the names of the committeemembers, the committees were usually inactive. In severalcases even those whose names were on the lists of committee membersdid not know of the existence of the committees, let alone thatthey were on them.

(Continues...)

Excerpted from Running Randomized Evaluations by RACHEL GLENNERSTER, KUDZAI TAKAVARASHA. Copyright © 2013 Princeton University Press. Excerpted by permission of PRINCETON UNIVERSITY PRESS.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

Edition details

• Publisher: Princeton University Press
• Publication date: 2013
• Language: English
• ISBN 10: 0691159270
• ISBN 13: 9780691159270
• Binding: Paperback
• Number of pages: 480