The reliability of replications: a study in computational reproductions

Nate Breznau, Eike Mark Rinke, Alexander Wuttke, Muna Adem, Jule Adriaans, Esra Akdeniz, Amalia Álvarez-Benjumea, Henrik Kenneth Andersen, Daniel Auer, Flávio Azevedo, Oke Bahnsen, Ling Bai, Dave Balzer, Paul Cornelius Bauer, Gerrit Bauer, Markus Baumann, Sharon Baute, Verena Benoit, Julian Bernauer, Carl Berning, Anna Berthold, Felix S. Bethke, Thomas Biegert, Katharina Blinzler, Johannes N. Blumenberg, Licia Bobzien, Andrea Bohman, Thijs Bol, Amie Bostic, Zuzanna Brzozowska, Katharina Burgdorf, Klaus Burger, Kathrin Busch, Juan Carlos Castillo, Nathan Chan, Pablo Christmann, Roxanne Connelly, Christian S. Czymara, Elena Damian, Eline A. de Rooij, Alejandro Ecker, S. Kellogg, Christina Eder, Maureen A. Eger, Simon Ellerbrock, Anna Forke, Andrea Förster, Danilo Freire, Christiaan Reinier Gaasendam, Konstantin Gavras, Vernon Gayle, Theresa Gessler, Timo Gnambs, Amélie Godefroidt, Max Grömping, Matthias Gross, Stefan Gruber, Tobias Gummer, Andreas Hadjar, Verena Halbherr, Jan Paul Heisig, Sebastian Hellmeier, Stefanie Heyne, Magdalena Hirsch, Mikael Hjerm, Oshrat Hochman, Jan H. Höffler, Andreas Hövermann, Sophia Hunger, Christian Hunkler, Nora Huth-Stöckle, Zsófia S. Ignácz, Sabine Israel, Laura Jacobs, Jannes Jacobsen, Bastian Jaeger, Sebastian Jungkunz, Nils Jungmann, Jennifer Kanjana, Mathias Kauff, Salman Khan, Sayak Khatua, Manuel Kleinert, Julia Klinger, Jan-Philipp Kolb, Marta Kołczyńska, John Kuk, Katharina Kunißen, Dafina Kurti Sinatra, Alexander Langenkamp, Robin C. Lee, Philipp M. Lersch, David Liu, Lea-Maria Löbel, Philipp Lutscher, Matthias Mader, Joan E. Madia, Natalia Malancu, Luis Maldonado, Helge MarahrensRoyal 2025 · DOI

TL;DR

This study reveals that computational reproducibility in social sciences is surprisingly unreliable, with errors stemming from both mistakes and procedural variations. Transparency, particularly access to original code, significantly boosts replication success, but it's not a silver bullet; better training and institutional support are crucial for improving scientific reliability.

What Problem Does This Paper Address?

The paper tackles the critical issue of computational reproducibility in scientific research, specifically within the social and behavioral sciences. While reproducibility is a cornerstone of scientific reliability, this study highlights that even seemingly straightforward tasks like computationally reproducing existing numerical results can be surprisingly difficult. The authors investigate why this is the case, exploring the variability among researchers attempting to replicate findings and identifying the sources of this uncertainty. This is crucial because a lack of reproducibility undermines the trustworthiness of scientific claims and hinders cumulative knowledge building.

Methods

The researchers designed an experiment involving 85 teams of 1-3 individuals tasked with computationally replicating the numerical results of a published study on immigration and public support for social policy. Participants were randomly assigned to one of two groups:

  • Transparent Group (TG): Received the original paper, its code, and a technical appendix.
  • Opaque Group (OG): Received only a description of the methods and no code.

Teams had three weeks to complete the replication. The study employed a mixed-methods approach, using both quantitative measures of replication success (directional reproduction, exact replication, replication error) and qualitative analysis of the teams' workflows to identify the sources of errors. They also developed "curated" and "trimmed" versions of the results to account for potential issues with the raw data and to better reflect real-world research scenarios.

Datasets

The study used the dataset from the paper:

  • **Brady D, Finnigan R.
  • Does Immigration Undermine Public Support for Social Policy?** This paper utilized two waves of International Social Survey Program (ISSP) data.

The researchers did not introduce a new benchmark dataset but rather used an existing, well-cited study with publicly available data and code as the basis for their replication experiment.

Key Insights and Ideas

  • Transparency is Key, But Not Sufficient: Access to the original study's code and detailed methodological information (the "transparent" condition) dramatically increased the success rate of computational reproductions. However, even with full transparency, perfect replication wasn't guaranteed.
  • "Push-Button" Replication is a Myth: The idea that computational replication can be a simple, automated process is an oversimplification. Researchers often need to understand the nuances of the original code and may encounter "tacit knowledge" that is difficult to explicate.
  • Errors are Common and Varied: Researchers made mistakes (coding errors, typos) and followed different procedural steps (recoding variables, handling missing data) than the original authors. These procedural variations, while not necessarily intentional mistakes, significantly impacted results.
  • Subjective Difficulty Matters: Researchers who perceived the replication task as more difficult were less likely to achieve successful reproductions. This suggests that the complexity of the task itself, or the perceived complexity, plays a significant role.
  • Software Choice Can Influence Results: While not a definitive cause of error, differences in software (e.g., Stata vs. R) and their default settings (like rounding) can contribute to discrepancies.

Key Results

  • Directional Reproduction:
    • Transparent Group (TG): 95.7% success rate.
  • Opaque Group (OG): 89.3% success rate.
  • Pooled (curated): 95.2% success rate.
  • Exact Numerical Reproduction (within 1%):
    • Transparent Group (TG): 76.9% success rate.
  • Opaque Group (OG): 48.1% success rate.
  • Only 14 teams (all in TG) achieved 100% exact numerical reproductions.
  • Team-Level Success (≥95% of models replicated):
    • Directional Reproduction: TG (79.5%) vs. OG (65.2%).
  • Exact Replication: TG (61.5%) vs. OG (6.5%).
  • Error Sources: "Procedural" errors were the most common (62 instances), followed by "Mistake" (29 instances).
  • Predictors of Error: Perceived difficulty and being in the opaque group significantly increased the likelihood of both "Mistake" and "Procedural" errors.

Limitations and Future Work

  • Sample Specificity: The study acknowledges that the specific dataset and variables used (socio-economic status, education categories) might have contributed to procedural variability. Future work could explore different types of data and research questions.
  • Population Inference: Drawing definitive conclusions about the entire population of social science researchers is challenging due to the difficulty in defining and sampling this population.
  • Motivation and Skill: While efforts were made to control for motivation (offering co-authorship), the study might not perfectly reflect the motivation levels of researchers in their day-to-day work. Follow-up studies could focus on experts in specific topics or disciplines.
  • Future Work: The authors suggest improved teaching of reproducibility, increased demands from institutions and journals for transparency (including code checking), and the development of more robust "push-button" replication environments. They also advocate for a more humble and cautious communication of scientific findings, acknowledging the inherent uncertainty.

Summary updated

Abstract

This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and <i>p</i>-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.
BibTeX
@inproceedings{breznau2025reliability,
  author = {Nate Breznau and Eike Mark Rinke and Alexander Wuttke and Muna Adem and Jule Adriaans and Esra Akdeniz and Amalia Álvarez-Benjumea and Henrik Kenneth Andersen and Daniel Auer and Flávio Azevedo and Oke Bahnsen and Ling Bai and Dave Balzer and Paul Cornelius Bauer and Gerrit Bauer and Markus Baumann and Sharon Baute and Verena Benoit and Julian Bernauer and Carl Berning and Anna Berthold and Felix S. Bethke and Thomas Biegert and Katharina Blinzler and Johannes N. Blumenberg and Licia Bobzien and Andrea Bohman and Thijs Bol and Amie Bostic and Zuzanna Brzozowska and Katharina Burgdorf and Klaus Burger and Kathrin Busch and Juan Carlos Castillo and Nathan Chan and Pablo Christmann and Roxanne Connelly and Christian S. Czymara and Elena Damian and Eline A. de Rooij and Alejandro Ecker and S. Kellogg and Christina Eder and Maureen A. Eger and Simon Ellerbrock and Anna Forke and Andrea Förster and Danilo Freire and Christiaan Reinier Gaasendam and Konstantin Gavras and Vernon Gayle and Theresa Gessler and Timo Gnambs and Amélie Godefroidt and Max Grömping and Matthias Gross and Stefan Gruber and Tobias Gummer and Andreas Hadjar and Verena Halbherr and Jan Paul Heisig and Sebastian Hellmeier and Stefanie Heyne and Magdalena Hirsch and Mikael Hjerm and Oshrat Hochman and Jan H. Höffler and Andreas Hövermann and Sophia Hunger and Christian Hunkler and Nora Huth-Stöckle and Zsófia S. Ignácz and Sabine Israel and Laura Jacobs and Jannes Jacobsen and Bastian Jaeger and Sebastian Jungkunz and Nils Jungmann and Jennifer Kanjana and Mathias Kauff and Salman Khan and Sayak Khatua and Manuel Kleinert and Julia Klinger and Jan-Philipp Kolb and Marta Kołczyńska and John Kuk and Katharina Kunißen and Dafina Kurti Sinatra and Alexander Langenkamp and Robin C. Lee and Philipp M. Lersch and David Liu and Lea-Maria Löbel and Philipp Lutscher and Matthias Mader and Joan E. Madia and Natalia Malancu and Luis Maldonado and Helge Marahrens},
  title = {{The reliability of replications: a study in computational reproductions}},
  booktitle = {Royal},
  year = {2025},
  doi = {10.1098/rsos.241038},
}
Figure