Researchers audited 17 non‑experimental American Economic Review papers (2013 and 2022/23) with alternative analyses vetted by independent experts. Only about 51% of these robustness tests stayed statistically significant, and average test statistics fell to roughly 70% of the originals. Economists surveyed overestimated robustness but could still pick which papers were most solid.
— Prestige economics findings are often fragile, so journals, media, and policymakers should demand robustness maps before relying on single studies.
Tyler Cowen
2025.08.30
100% relevant
Douglas Campbell et al.’s study of 17 AER papers showing 0%–93% robustness significance rates (mean 51%) and reduced t/z-values (mean 70%).
← Back to All Ideas