The Unreasonable Effectiveness of Open Science in AI: A Replication Study

Odd Erik Gundersen, Odd Cappelen, Martin Mølnå, Nicklas Grimstad NilsenAAAI 2025 · DOI

Abstract

Reproducibility is a cornerstone of the scientific method, yet the field of Artificial Intelligence frequently struggles with the verification of published results. This study evaluates 30 highly cited empirical AI papers from 2012, 2014, and 2016 to quantify the impact of open science practices on reproducibility. Using a standardized 40-hour replication protocol, the authors found that the availability of both code and data is the most significant predictor of success, yielding an 86% success rate. Conversely, studies providing only data saw success rates drop to 33%. The analysis identifies specific documentation failures, particularly regarding dataset partitioning and versioning, as primary barriers to replication, while suggesting that code documentation quality is less critical than the mere presence of the code itself.

Introduction

In the rapidly evolving landscape of Artificial Intelligence, "state-of-the-art" claims are often accepted without independent verification. This "reproducibility crisis" stems from the complexity of modern AI systems, which often rely on specific hyperparameters, private datasets, or undocumented preprocessing steps. While the field has moved toward more open practices, the actual effectiveness of these measures remains under-studied. This paper presents a meta-analysis of 30 influential AI studies to identify which factors most strongly correlate with successful independent replication.

Methodology

The researchers selected the top 10 most-cited empirical AI papers from three distinct years (2012, 2014, and 2016). To ensure a realistic and consistent evaluation, they employed a structured methodology:

  1. Classification: Papers were categorized into four levels of reproducibility: R1 (paper only), R2 (paper + code), R3 (paper + data), and R4 (paper + code + data).
  2. The 40-Hour Protocol: A team of researchers spent a maximum of 40 focused hours attempting to replicate each study, simulating the effort an average researcher might invest.
  3. Problem Cataloging: The team identified 20 distinct problem types (P1–P20) encountered during the process, ranging from ambiguous method descriptions to hardware limitations.
  4. Statistical Analysis: Logistic regression was used to determine which of these problems were the strongest predictors of replication failure.

Results and Key Insights

The study revealed a stark contrast in success rates based on the artifacts provided by the original authors. The most significant finding was the "unreasonable effectiveness" of providing both code and data (R4).

Replication success rates categorized by reproducibility type (R1, R3, R4).

As shown in the data, the jump from R3 (data only) to R4 (code + data) represents a massive increase in reproducibility—from 33% to 86%. This suggests that code is not merely a supplement but the primary engine of scientific verification in AI.

The Documentation Paradox

Interestingly, the study found that while high-quality data documentation is critical, the quality of code documentation matters significantly less. If the source code is available, skilled researchers can typically reverse-engineer the implementation details. However, if the data is poorly specified, the replication effort often fails regardless of the code's quality.

Analysis of Failure Modes

To understand why replications fail, the authors mapped specific hurdles to a catalog of 20 problem types.

Detailed catalog of the 20 problem types (P1–P20) identified during the study.

The logistic regression highlighted three critical "deal-breakers" that most frequently led to failure:

  • P15 (Dataset Mismatch): Discrepancies between the dataset version specified in the paper and the version available online.
  • P18 (Partitioning Information): Missing details on how data was split into training, validation, and test sets.
  • P7 (Random Seeds): Unspecified random number generators, which can lead to significant variance in results.

Discussion and Limitations

While the results provide a clear roadmap for improving AI research practices, the study has several limitations. The sample size of 30 papers is relatively small, and the focus on highly cited papers may introduce a selection bias toward more robust studies. Furthermore, the 40-hour limit may be insufficient for complex deep learning models that require extensive setup.

Despite these limitations, the study underscores a clear mandate for the AI community: sharing code and data is the single most effective way to ensure scientific integrity. Future research should examine whether the recent introduction of reproducibility checklists at major conferences has successfully mitigated these common pitfalls.

Summary updated

BibTeX
@inproceedings{gundersen2025unreasonable,
  author = {Odd Erik Gundersen and Odd Cappelen and Martin Mølnå and Nicklas Grimstad Nilsen},
  title = {{The Unreasonable Effectiveness of Open Science in AI: A Replication Study}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i25.34818},
}
Figure