The Efficacy of RCTs: Actual Empirics
What’s the empirical evidence that RCTs actually improve policy? Incisive comments from the noble Lant Pritchett:
The argument that RCTs would be more than a tiny component of the overall process of improving development outcomes seems, even now, 10 years in, at best not provable and at worst not very likely—as it is at odds with some basic known facts about development and about policy formulation and implementation.
What “basic known facts” does he have in mind? First, RCTs can’t be crucial for development, because many countries became highly developed long before RCTs mattered.
First of all, the argument that RCTs had, until recently, been used sparingly, if at all, and yet are important in achieving good outcomes sits in kind of embarrassing counterpoint with the obvious fact that lots of countries have really good outcomes. That is whether one uses the Human Development Index or the OECD Better Life Index or any social indicator—from poverty to education to health to life satisfaction—there is a similar set of countries near the top… No one has ever made the arguments that these countries are developed and prosperous because they used rigorous evidence—much less RCTs—in formulating policy and programs. While one might have faith that RCTs can help along the path to development, RCTs didn’t help for those that are there now.
Second, the highest observed economic growth in human history happened in the decades right before RCTs became fashionable:
Second, at about the time (early 2000s) that the randomista movement, which often claimed to be about reducing poverty, was gaining steam, several countries were experiencing or had experienced the rapidest reductions in low-bar absolute poverty in the history of man. Indonesia, China, Vietnam had all seen dollar-a-day–like poverty measures fall from well over half of their population to under 20 percent in less than 30 years. In Vietnam the World Bank figures show dollar-a-day poverty falling from over 40 percent to 16.9 percent just between 2002 and 2008. If these countries did this completely without any use of RCTs (and they did), then from where exactly does the argument that using RCTs will accelerate poverty reduction come? Is there any evidence that countries that used more RCTs had more rapid poverty reduction than those that didn’t? No. While it might be the case that RCTs could accelerate poverty reduction this was (and is) a faith-based, not evidence-based, claim.
Third, the past is the best predictor of the future – and past RCTs had little influence on policy:
Third, there was the recent historical experience with RCTs in social policy in the USA. The randomistas were not proposing new methods or techniques but rather broader adoption into the field of development methods that already had a long history. There was a big fad toward the use of experiments in a variety of social policy domains in the USA in the 1970s. The Rand Health Insurance Experiment, carried out from 1971 to 1986, is still the largest health-policy experiment ever done. Income maintenance (negative income tax) experiments were carried out between 1968 and 1982 in four sites around the USA. A 1970 act of Congress authorized experiments such as the Housing Allowance Demand Experiment. The Kansas City Police Preventive Patrol experiment was carried out in 1972–73. This created ample capacity in the USA for doing social experiments inside organizations like Rand, Abt Associates, and Mathematica Policy Research, among others. At the founding of J-PAL there was 35 years of experience with RCTs in the USA on which to draw evidence of their efficacy (or not) as tools for improving policy and hence human outcomes. You would have thought that if the widespread use of RCTs had strongly impacted policy in the USA, this would have been put forward as evidence. Strangely, whether or not decades of social policy RCTs actually did have impact on policies and outcomes in the USA just kind of never came up in arguing that they would in developing countries.
What’s the alternative? Pritchett elsewhere argues that orthodox growth economics worked wonders for humanity but got almost no credit. In fact, it got turned into the bizarre conspiracy theory of “neoliberalism” and denounced by left and right alike. Fortunately, as Vanilla Sky teaches us, “Every passing minute is another chance to turn it all around.” Growth economists have made plenty of mistakes, but they should admit that RCTs are a trendy distraction and pick up where they left off.
The post The Efficacy of RCTs: Actual Empirics appeared first on Econlib.