I uploaded this post to ChatGPT and asked for counter-examples ("paragon" papers) to see if the paper I had in mind ("Robert Axelrod’s Evolution of Cooperation tournament results 1981") would be in the response. Sure enough, it was listed there (along with several often-cited popular studies). Thoughts?
It's worth noting –though I think you were already hinting at it– that since no paper is perfect, Noah and others who follow his advice will tend to see more of the epistemic flaws in papers that don't confirm their biases, and overlook the flaws in those that do confirm theirs.
Fascinating essay, BC. I knew there was a really good reason that I abhor keeping up with "the literature" in economics. I much prefer reading well reasoned essays on the 'stack from authors like you and John H. Cochrane and David Friedman and Arnold Kling and a few others whose names slip my mind just this second.
I agree with your first statement. I disagree with your second statement, even though math is a wonderful and useful tool for economists, just as it is for other sciences. Sadly enough, heavy reliance on math models instead of cogent verbal models is one of the shortcomings of many economists, in my opinion.
I'm with Noah Smith on this one - I think you're thinking he's expecting too much. What you're calling a 0.2 might be good enough for him to investigate further - his point is, if someone is trying to convince you have a paradigm, have them give you the best from it, which should be better than 0.05. There are so many papers and paradigms around now that if you don't, you'll get caught in the mud moat. There's always a chance there's something there, but probably better to play the odds.
The mantra should be "No *empirical* paper is that good". Circumscribing the claim in this way seems important for it's correctness.
Theory papers are different. They really can introduce a conceptual framework, say all they need to say, then leave. The Use of Knowledge in Society, for instance, really is that good. The Market for Lemons, The Nature of the Firm and A Difficulty in the Concept of Social Welfare might be other examples from economics. Claude Shannon's master's thesis is the proverbial example.
Is this a phenomenon about economics in particular? Or perhaps about fields that do large empirical work vs inventing new models that can be validated cheaply?
I can pretty easily cite 1-2 definitive papers in fields I know well, for example:
I uploaded this post to ChatGPT and asked for counter-examples ("paragon" papers) to see if the paper I had in mind ("Robert Axelrod’s Evolution of Cooperation tournament results 1981") would be in the response. Sure enough, it was listed there (along with several often-cited popular studies). Thoughts?
Although I often learn from Smith, on any point of contention, I have never found him more persuasive than Caplan. This is such an example.
It's worth noting –though I think you were already hinting at it– that since no paper is perfect, Noah and others who follow his advice will tend to see more of the epistemic flaws in papers that don't confirm their biases, and overlook the flaws in those that do confirm theirs.
Fascinating essay, BC. I knew there was a really good reason that I abhor keeping up with "the literature" in economics. I much prefer reading well reasoned essays on the 'stack from authors like you and John H. Cochrane and David Friedman and Arnold Kling and a few others whose names slip my mind just this second.
If your economic proposition can't be stated fully and clearly without mathematics, I question its relevance.
If your economic proposition can't be meaningfully demonstrated or depicted with mathematics, I question its validity.
I made that up.
I agree with your first statement. I disagree with your second statement, even though math is a wonderful and useful tool for economists, just as it is for other sciences. Sadly enough, heavy reliance on math models instead of cogent verbal models is one of the shortcomings of many economists, in my opinion.
I'm with Noah Smith on this one - I think you're thinking he's expecting too much. What you're calling a 0.2 might be good enough for him to investigate further - his point is, if someone is trying to convince you have a paradigm, have them give you the best from it, which should be better than 0.05. There are so many papers and paradigms around now that if you don't, you'll get caught in the mud moat. There's always a chance there's something there, but probably better to play the odds.
The mantra should be "No *empirical* paper is that good". Circumscribing the claim in this way seems important for it's correctness.
Theory papers are different. They really can introduce a conceptual framework, say all they need to say, then leave. The Use of Knowledge in Society, for instance, really is that good. The Market for Lemons, The Nature of the Firm and A Difficulty in the Concept of Social Welfare might be other examples from economics. Claude Shannon's master's thesis is the proverbial example.
Is this a phenomenon about economics in particular? Or perhaps about fields that do large empirical work vs inventing new models that can be validated cheaply?
I can pretty easily cite 1-2 definitive papers in fields I know well, for example:
For LLMs: the Attention paper (https://arxiv.org/pdf/1706.03762) and the Chinchilla paper (https://arxiv.org/pdf/2203.15556)
For image generation models: the diffusion paper (https://arxiv.org/pdf/2006.11239
For phylogenetics: Felsenstein 81 (http://gen-ftp.princeton.edu/sysbio/Felsenstein81.pdf)