Introduction
The "replication crisis" has fundamentally challenged the reliability of findings in the social sciences over the last decade. While previous large-scale efforts, such as the Reproducibility Project: Psychology (RPP), reported replication rates as low as 36%, critics argued these efforts lacked the statistical power to detect real but smaller effects. This research investigates the "gold standard" of scientific publishing—Nature and Science—to determine if high-impact social science findings hold up under rigorous, high-powered replication protocols.
Methodology
The study targeted 21 experimental social science papers published between 2010 and 2015. To ensure maximum reliability, the researchers employed a two-stage replication process. Stage 1 was powered at 90% to detect 75% of the original effect size. If replication failed, Stage 2 increased the sample size to achieve 90% power for 50% of the original effect. On average, replication samples were five times larger than the originals.
Key methodological safeguards included:
- Pre-registration: All analysis plans were made public before data collection to prevent "p-hacking."
- Author Collaboration: Original authors provided materials and protocols to ensure faithful replications.
- Prediction Markets: Peer researchers bet on replication outcomes to gauge community consensus before results were known.
Key Findings: The "Shrinkage" Effect
The study found a replication rate of 61.9% (13 out of 21 studies). While higher than previous large-scale efforts, the results revealed a significant "shrinkage" in effect sizes. The mean standardized effect size in replications was 0.249, roughly 50% of the 0.460 reported in the original papers.
Even among successful replications, effect sizes were approximately 75% of the original values. For failed replications, the effect sizes were near zero. This suggests that publication bias and the "winner's curse"—where journals prioritize novel, extreme results—lead to substantial overestimations of effect sizes in top-tier journals.
The Wisdom of the Crowd
One of the most striking results was the accuracy of prediction markets. Peer beliefs (averaging 63.4%) were remarkably close to the actual replication rate (61.9%). This indicates that the scientific community often possesses an intuitive sense of which findings are robust and which are likely "shaky," despite their publication in prestigious venues.
Conclusion and Limitations
While high-impact journals show better replicability than the broader field, the significant reduction in effect sizes remains a concern. The authors acknowledge that while high power is essential, it is not a panacea for fixing reproducibility if underlying effects are non-existent or heavily inflated by original reporting biases. Future work will involve pooling data from multiple large-scale projects to identify the specific determinants of replication success.
Summary updated