TL;DR
This study reveals that computational reproducibility in social sciences is surprisingly unreliable, with errors stemming from both mistakes and procedural variations. Transparency, particularly access to original code, significantly boosts replication success, but it's not a silver bullet; better training and institutional support are crucial for improving scientific reliability.
What Problem Does This Paper Address?
The paper tackles the critical issue of computational reproducibility in scientific research, specifically within the social and behavioral sciences. While reproducibility is a cornerstone of scientific reliability, this study highlights that even seemingly straightforward tasks like computationally reproducing existing numerical results can be surprisingly difficult. The authors investigate why this is the case, exploring the variability among researchers attempting to replicate findings and identifying the sources of this uncertainty. This is crucial because a lack of reproducibility undermines the trustworthiness of scientific claims and hinders cumulative knowledge building.
Methods
The researchers designed an experiment involving 85 teams of 1-3 individuals tasked with computationally replicating the numerical results of a published study on immigration and public support for social policy. Participants were randomly assigned to one of two groups:
- Transparent Group (TG): Received the original paper, its code, and a technical appendix.
- Opaque Group (OG): Received only a description of the methods and no code.
Teams had three weeks to complete the replication. The study employed a mixed-methods approach, using both quantitative measures of replication success (directional reproduction, exact replication, replication error) and qualitative analysis of the teams' workflows to identify the sources of errors. They also developed "curated" and "trimmed" versions of the results to account for potential issues with the raw data and to better reflect real-world research scenarios.
Datasets
The study used the dataset from the paper:
- **Brady D, Finnigan R.
- Does Immigration Undermine Public Support for Social Policy?** This paper utilized two waves of International Social Survey Program (ISSP) data.
The researchers did not introduce a new benchmark dataset but rather used an existing, well-cited study with publicly available data and code as the basis for their replication experiment.
Key Insights and Ideas
- Transparency is Key, But Not Sufficient: Access to the original study's code and detailed methodological information (the "transparent" condition) dramatically increased the success rate of computational reproductions. However, even with full transparency, perfect replication wasn't guaranteed.
- "Push-Button" Replication is a Myth: The idea that computational replication can be a simple, automated process is an oversimplification. Researchers often need to understand the nuances of the original code and may encounter "tacit knowledge" that is difficult to explicate.
- Errors are Common and Varied: Researchers made mistakes (coding errors, typos) and followed different procedural steps (recoding variables, handling missing data) than the original authors. These procedural variations, while not necessarily intentional mistakes, significantly impacted results.
- Subjective Difficulty Matters: Researchers who perceived the replication task as more difficult were less likely to achieve successful reproductions. This suggests that the complexity of the task itself, or the perceived complexity, plays a significant role.
- Software Choice Can Influence Results: While not a definitive cause of error, differences in software (e.g., Stata vs. R) and their default settings (like rounding) can contribute to discrepancies.
Key Results
- Directional Reproduction:
- Transparent Group (TG): 95.7% success rate.
- Opaque Group (OG): 89.3% success rate.
- Pooled (curated): 95.2% success rate.
- Exact Numerical Reproduction (within 1%):
- Transparent Group (TG): 76.9% success rate.
- Opaque Group (OG): 48.1% success rate.
- Only 14 teams (all in TG) achieved 100% exact numerical reproductions.
- Team-Level Success (≥95% of models replicated):
- Directional Reproduction: TG (79.5%) vs. OG (65.2%).
- Exact Replication: TG (61.5%) vs. OG (6.5%).
- Error Sources: "Procedural" errors were the most common (62 instances), followed by "Mistake" (29 instances).
- Predictors of Error: Perceived difficulty and being in the opaque group significantly increased the likelihood of both "Mistake" and "Procedural" errors.
Limitations and Future Work
- Sample Specificity: The study acknowledges that the specific dataset and variables used (socio-economic status, education categories) might have contributed to procedural variability. Future work could explore different types of data and research questions.
- Population Inference: Drawing definitive conclusions about the entire population of social science researchers is challenging due to the difficulty in defining and sampling this population.
- Motivation and Skill: While efforts were made to control for motivation (offering co-authorship), the study might not perfectly reflect the motivation levels of researchers in their day-to-day work. Follow-up studies could focus on experts in specific topics or disciplines.
- Future Work: The authors suggest improved teaching of reproducibility, increased demands from institutions and journals for transparency (including code checking), and the development of more robust "push-button" replication environments. They also advocate for a more humble and cautious communication of scientific findings, acknowledging the inherent uncertainty.
Summary updated