RCTs and the Status Quo: The Special Relationship
In economics, Randomized Controlled Trials (RCTs) now stand at the pinnacle of the methodological hierarchy. “Natural experiments” are a distant second. Work based on old-fashioned observational data is actually hard to publish anywhere prestigious. For many scholars, RCTs aren’t just the gold standard of research. Nothing else is even fungible. This is the Age of the Randomista.
Which raises a serious problem: How can researchers address questions where no RCT is feasible? To do an RCT on national monetary policy, for example, you would have to randomly assign monetary policies to a bunch of countries. Not gonna happen. To do an RCT on national disincentive effects of welfare, you would have to randomly assign welfare policies to a bunch of countries. Again, not gonna happen.
Sure, you could run some RCT laboratory experiments on monetary policy. But why assume that some silly games in a lab carry over into the real world? Similarly, you could run a pilot welfare program for a single city and measure the effects. But perhaps a lot of the labor supply response comes from the society-wide erosion of stigma against idleness. If so, your pilot program will fail to detect it.
The same goes if a Marxist claims that once capitalism has been eliminated, people will work for the sheer joy of contributing to the community. You could try running an experiment on a utopian commune, but the Marxist could protest, “Capitalism must be eliminated world-wide before my claim holds.” And on the flip side, the collapse of Communism wasn’t based on RCTs either. Critics just said, “This is an awful system and must be dismantled.” And amazingly managed to get their way in a bunch of countries.
The upshot is that, like the US and UK, RCT methodology and the status quo (SQ) have a “special relationship.” If you take RCTs seriously, you have to label virtually any radical departure from the SQ as “unscientific.” After all, if the change is radical, it won’t be feasible to run an RCT. And if RCT is the only scientifically respectable methodology, then every radical departure from the SQ is scientifically baseless.
Ironic, given all the publicity about how wonderfully “radical” the RCTers are.
Still, there are multiple ways to interpret the special relationship between RCT and SQ. Let’s start with the mildest, then intensify, step by step.
1. Big changes from the SQ can’t be justified using RCTs, but neither are they unjustified. As far as social science is concerned, it’s a “Whereof one cannot speak, thereof one must be silent” situation. In other words, RCTs tell us to be agnostic about big deviations from the SQ.
2. Big changes from the SQ can’t be justified using RCTs, so we should, by default, be skeptical. That’s how the FDA would react to new untested pharmaceuticals, right?
3. Big changes from the SQ can’t be justified using RCTs, so we should expect them to fail – and regard their proponents as charlatans. Even if, by some miracle, they happen to be right, their methodology is reckless.
4. Big changes from the SQ can’t be justified using RCTs, so we should expect them to be disastrous – and regard their proponents as monsters. You would practically have to be a psychopath to blindly push for big social changes.
If you are a staunch RCT person, however, positions 2, 3, and 4 all suffer from a common problem: None of them has ever been justified by an RCT! There has never been an RCT showing that big changes sans RCTs merit skepticism. There has never been an RCT showing that big changes sans RCTs typically fail. And there has never been an RCT showing big changes sans RCTs typically end in disaster. Indeed, as far as I know, there aren’t even any old-fashioned observational studies supporting these conclusions.
Should we then retreat to position 1? It too suffers from a dire problem. Namely: If you don’t believe that changes supported by RCTs are, on average, better than changes not supported by RCTs, why do you support RCTs in the first place?
In short, there’s a dilemma of methodological advocacy: You can be enthusiastic, or you can apply the methodology. But not both. The randomista crusade is either hypocritical or stillborn.
Is there any way to escape from this dilemma? Yes, but only with repentant methodological humility. Admit that the real foundation of science is just common sense. And common sense tells us that RCTs are the most helpful way to advance our understanding of extremely narrow questions. But when you ponder bigger questions, RCTs are just one intellectual input out of many. Including the bigger question of, “When should we dismiss people who fail to use RCTs as charlatans?” For pharmaceuticals, the right answer is “often.” For economic growth, in contrast, the charlatans are those who dismiss everything we’ve learned without RCTs.
The post appeared first on Econlib.