Introduction
In the scientific method, the ability for an independent team to replicate a result is the bedrock of establishing a proven fact. Without replication, a finding remains a "one-off" observation rather than a generalizable truth. Despite the digital nature of Artificial Intelligence (AI) research—which theoretically facilitates seamless sharing of code and data—the field has long struggled with a lack of transparency. This study quantifies the "reproducibility crisis" by surveying 400 papers from top-tier conferences, the International Joint Conference on AI (IJCAI) and the Association for the Advancement of AI (AAAI), to determine if current documentation practices meet the standards required for scientific validation.
Defining Reproducibility in AI
The authors propose a formal framework that categorizes reproducibility into three distinct degrees, moving from specific implementations to general theories:
- R1: Experiment Reproducible: Achieving the same results using the original code and data. This is the most stringent level, requiring exhaustive documentation of the experimental environment.
- R2: Data Reproducible: Achieving the same results with a different implementation but the same data. This tests whether the described method is robust beyond a single codebase.
- R3: Method Reproducible: Achieving the same results with a different implementation and different data. This represents the ultimate goal of AI research: proving a generalizable theory.
A critical distinction is made between the AI Program (the specific implementation) and the AI Method (the conceptual algorithm). True scientific progress relies on proving the Method, yet the documentation required to even reproduce the Program (R1) is often missing.
Methodology
To assess the state of the field, the researchers identified key variables across three factors—Method, Data, and Experiment—that are essential for reproduction. These include pseudo-code, hardware specifications, data splits, and hyperparameter settings. They manually audited 400 papers from IJCAI (2013, 2016) and AAAI (2014, 2016), checking for the explicit presence of these variables.
The Documentation Gap
The findings reveal a significant shortfall in documentation. Not a single paper out of the 400 surveyed provided all the variables necessary to satisfy any of the three "R" levels. Average reproducibility scores hovered between 0.24 and 0.26 on a scale of 0 to 1, indicating that only about a quarter of the necessary information is typically shared in a published paper.
As shown in the data, while some variables like pseudo-code (54%) and problem definitions (47%) are relatively common, others are critically under-reported. For instance, hardware specifications were only present in 27% of papers, and experimental setup details were documented in 69% of cases. Data sharing emerged as a relative bright spot, with approximately 49% of empirical papers providing access to their datasets.
Trends and Progress
While the overall scores remain low, the study found evidence of gradual improvement. There was a statistically significant increase in the R1D metric (experiment documentation) between the earlier (2013/14) and later (2016) conference installments.
This upward trend suggests that the AI community is becoming more diligent about describing their experimental procedures, even if the sharing of actual code and comprehensive environment details has not yet seen a similar surge.
Conclusion and Future Outlook
The study highlights a "Big Zero": the current inability to fully reproduce results from top-tier AI literature based solely on the published documentation. While the manual nature of this review has limitations—it may miss information standard to specific sub-fields—it provides a necessary baseline for the community.
The authors suggest that these reproducibility metrics could serve as a checklist for conference reviewers and journal editors. By institutionalizing these standards, the field can move toward a future where AI research is not just a collection of impressive "one-offs," but a transparent and verifiable science.
Summary updated