Esther Duflo is a bit of a hero of mine.
She has done pioneering work as a founder and director of the Abdul Latif Jameel Poverty Action Lab (J-PAL) based out of MIT. Not to mention she just won the John Bates Clark medal for being "the American economist under 40 who is judged to have made the most significant contribution to economic thought and knowledge". So what exactly is all the fuss about?
Seeing what works
The problem with humanitarian interventions is that they’re often not as rigorously evaluated as they should be, so it’s difficult to see how much of a difference is being made, if at all, or whether impact is simple due to other outside forces. The point of J-PAL is to conduct impact evaluations that are able to see exactly that.
How do they do that?
Researchers set up science-like experiments called randomized controlled trials (RCTs) with a control group and a treatment group, usually across different villages in the same region. They monitor the changes in the treatment group after some type of humanitarian intervention and compare them to the control group which represents the state of nature sans intervention. These are the types of trials one normally associates with the medical field, but here they are being applied towards a different end.
Thinking small to think big
The point of J-PAL’s research is to focus on very specific issues that can then be used to shape policies. These targets range from the best way to increase malarial bed net usage to how to get children immunized. They want to avoid getting lost on the “big” questions and center effort upon which interventions get the most bang for the buck on a micro level. Check out Duflo’s TED talk where she explains what J-PAL is specifically trying to accomplish.
Some findings across different countries?
“Impersonally administered and direct incentives” are the most effective way to increase teacher attendance. And “even very small user fees” discourage preventative healthcare measures. You can check out more findings here.
Yes, but…
Despite the overwhelming praise, there are some common critiques that J-PAL often has to counter.
What about the control groups who don’t receive the intervention?
In a Development Drums podcast, Rachel Glennerster, Executive Director of J-PAL, explains that they work with NGOs who have limited resources anyways, so the same number of people receive the intervention as otherwise would have—J-PAL just changes the distribution of the goods or services.
Now for scalability—we can’t actually expect something that worked in Malawi to work in India and vice versa?
When talking about biological issues like intestinal worms, some conclusions can be fairly universal. Those that are based on behavior are harder to bridge. But J-PAL is expanding in order to test “solutions” they’ve found in some areas in places that are wildly different to see if they carry over. This process obviously takes some time so J-PAL needs to be careful about making sweeping generalizations for the sake of policy.
In the end, aren’t they just experimenting with human beings?
Arguably, humanitarian interventions are always best guesses and experiments until we can actually observe their impact. Just because we think something should work doesn’t mean that it will (and it often doesn’t). Since NGOs and governments will continue to implement certain programs and policies no matter what, RCTs can at least measure whether they are actually accomplishing their goals. In Duflo's words, “With precautions in mind, we believe that not experimenting is what is unethical. If you scale up a policy that does no good, or even some unintended harm, or if you do not adopt a policy that could make a great amount of good, this is when you are carrying out gigantic experiments on people's lives. And that should not be acceptable.”
RCTs can’t accomplish everything—for one, they obviously can’t capture qualitative project evaluations. And though they are costly because they require high technical proficiency and carefully designed conditions, we might begin to see a surplus of researchers on this issue that pulls the attention from other alternatives. That is, if researchers begin to frame their approach to questions in the same way, creativity will lose out and we will only see one approach where there should be many. (And the technical debate in the economics world over RCTs still continues.)
In spite of the critiques, work from J-PAL and Duflo has certainly revolutionized the field of development in the past few years, so it’s good for us to get a clear picture of what they’re doing and where the limitations of RCTs lie. It’s definitely one great way to make sure development is done right.
Hi Maria, thanks for the post on RCT’s - I have to admit this is the first time I am coming across these. My question is (10:30 of the TED talk) is whether making people used to getting things for free or with some extra gifts is sustainable and leads people to non corrupt ways? Isn’t the way of thinking presented a little bit simplistic and feasible only in some of the many cases?
Once again, we are facing the complexity of the development and the undeniable role the scientists need to play in that field.
Thanks Maria - there is one more thing that is overlooked here, understandably, because Esther is an economist. And that is the fact that we are dealing with biological systems here. And these change over time, they evolve.
I once attended a talk of Jeffrey Sachs where he said ‘What the seatbelt is to the developed world, should be the bednet for the developing world’. A nice statement, but he forgot entirely that a seatbelt will still work 50 years from now, the bednets won’t because mosquitoes become resistant to the insecticides they’re impregnated with… So it goes for drugs, where pathogens develop resistance. Long-term projections on benefits can therefore be misleading because the strategy falters mid-way…
Simple solutions usually work best. I’m just back from a Development Cooperation Conference, where Polish Foreign Ministry rep helplessly said (responding the question from the audience) that there are NO aid efficiency evaluation mechanisms. Education at the top strongly needed!
Thanks Maria for your post! Actually, I didn’t not about this project, however I also believe that simple solutions usually work best, as also Robert said! Thanks again!
@Helena: The conventional wisdom was, for a long time, that if we don’t charge for products then it will create dependency because there will be neither value associated to them nor incentives to use them properly. But the conclusion that J-PAL came to here was a “hand up, not a hand out,” meaning that individuals or households do assign value to the products but simply can’t afford them.
However, I wonder how this value varies with the degree to which individuals actually observe improvements in their lives as a result of a product—because perhaps that is a prerequisite for creating value. In that case, if the effects aren’t very obvious then, like you said, that might cause this approach to work only in certain cases. Can you think of any other examples where that philosophy might not work?
@Bart: Thanks so much for your insight. That’s an excellent point. Maybe then a more appropriate conclusion would be that eliminating costs to preventative healthcare measures encourages their short term usage, the caveat being that their ultimate success depends upon adapting the appropriate healthcare methods over time. Again, thank you for pointing that out.
@Robert & Hussam: Thanks! I think the appeal of simple solutions is that they are easily controlled and observed. It’s harder to see the specific ramifications of big solutions because they’re so, well, broad. (And @Robert, oh my, I hope the ministry rep just misspoke!)
I got to this TED talk too and this is a good one, a nice example of simplifying things and then trying them out adapting and having a result. I want to say that she points out lack of aid adaption.
A couple of points here -
RCT’s are a statistical tool to isolate causality. They tell us what works and gives an extremely clean estimate of an estimated size of the intervention. They don’t really tell us how it works - we observe pre-and post intervention general equilibrium values. We don’t observe what has happened in between, aside from the provision of say more textbooks… we need to know how the communities respond to the treatment in order to understand the findings.
second, even with RCT’s, many questions are still unanswered. look at the link to teacher attendance (which i’m working on at the moment)
http://www.povertyactionlab.org/policy-lessons/education/teacher-attendance
Evaluations 1 & 4 work (though paper 1 is misleading), while 5 and 6 don’t work. same with the incentives. The trouble is we conclude “yes it works” or “no it doesn’t” with no idea why. on the face of it, all are similar interventions - so would we expect similar effects? is it the change in treatment or change in scenario which matters?
Deaton (2010) sums it all up perfectly - by searching for a precise estimate of the Average Treatment Effect, we are eliminating the very information which is of most interest.
RCT’s are an amazing way of giving precise point estimates to measure the average impact of a program, which is itself a wonderful step. It allows a very simple method for evaluation, which should improve M&E in aid projects. However, it should be viewed as simply another statistical tool for economists to measure interventions. Not everything in live can be randomised, and while (to paraphrase) it’s good to take the con out of econometrics, it’d be a shame to take the econ out also..
Thanks for your excellent points, Paul. Sorry for not going into too much of the “black magic” here but you sum the issues up well. The trouble lies in discovering the “how” factor since that would capture the true effect of the intervention and allow for better replication (or tell us that it’s impossible). RCTs are great but at the end of the day they are one statistical tool alongside an array of other evaluation methods because they focus on making something work without explaining the precise causality. Best of luck with your research!