Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Colin F. Camerer, Anna Dreber, Felix Holzmeister, Teck‐Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A. Nosek, Thomas Pfeiffer, Adam Altmejd, Nick Buttrick, Taizan Chan, Yiling Chen, Eskil Forsell, Anup Gampa, Emma Heikensten, Lily Hummer, Taisuke Imai, Siri Isaksson, Dylan Manfredi, Julia Rose, Eric‐Jan Wagenmakers, Hang WuNature Human Behaviour 2018 · DOI

Introduction

The "replication crisis" has fundamentally challenged the reliability of findings in the social sciences over the last decade. While previous large-scale efforts, such as the Reproducibility Project: Psychology (RPP), reported replication rates as low as 36%, critics argued these efforts lacked the statistical power to detect real but smaller effects. This research investigates the "gold standard" of scientific publishing—Nature and Science—to determine if high-impact social science findings hold up under rigorous, high-powered replication protocols.

Methodology

The study targeted 21 experimental social science papers published between 2010 and 2015. To ensure maximum reliability, the researchers employed a two-stage replication process. Stage 1 was powered at 90% to detect 75% of the original effect size. If replication failed, Stage 2 increased the sample size to achieve 90% power for 50% of the original effect. On average, replication samples were five times larger than the originals.

Key methodological safeguards included:

  • Pre-registration: All analysis plans were made public before data collection to prevent "p-hacking."
  • Author Collaboration: Original authors provided materials and protocols to ensure faithful replications.
  • Prediction Markets: Peer researchers bet on replication outcomes to gauge community consensus before results were known.

Key Findings: The "Shrinkage" Effect

The study found a replication rate of 61.9% (13 out of 21 studies). While higher than previous large-scale efforts, the results revealed a significant "shrinkage" in effect sizes. The mean standardized effect size in replications was 0.249, roughly 50% of the 0.460 reported in the original papers.

Relative standardized effect sizes for the 21 replication attempts. Blue diamonds indicate successful replications, while orange diamonds indicate failures to replicate the original findings.

Even among successful replications, effect sizes were approximately 75% of the original values. For failed replications, the effect sizes were near zero. This suggests that publication bias and the "winner's curse"—where journals prioritize novel, extreme results—lead to substantial overestimations of effect sizes in top-tier journals.

The Wisdom of the Crowd

One of the most striking results was the accuracy of prediction markets. Peer beliefs (averaging 63.4%) were remarkably close to the actual replication rate (61.9%). This indicates that the scientific community often possesses an intuitive sense of which findings are robust and which are likely "shaky," despite their publication in prestigious venues.

Conclusion and Limitations

While high-impact journals show better replicability than the broader field, the significant reduction in effect sizes remains a concern. The authors acknowledge that while high power is essential, it is not a panacea for fixing reproducibility if underlying effects are non-existent or heavily inflated by original reporting biases. Future work will involve pooling data from multiple large-scale projects to identify the specific determinants of replication success.

Summary updated

BibTeX
@inproceedings{camerer2018evaluating,
  author = {Colin F. Camerer and Anna Dreber and Felix Holzmeister and Teck‐Hua Ho and Jürgen Huber and Magnus Johannesson and Michael Kirchler and Gideon Nave and Brian A. Nosek and Thomas Pfeiffer and Adam Altmejd and Nick Buttrick and Taizan Chan and Yiling Chen and Eskil Forsell and Anup Gampa and Emma Heikensten and Lily Hummer and Taisuke Imai and Siri Isaksson and Dylan Manfredi and Julia Rose and Eric‐Jan Wagenmakers and Hang Wu},
  title = {{Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015}},
  booktitle = {Nature Human Behaviour},
  year = {2018},
  doi = {10.1038/s41562-018-0399-z},
}
Figure