The modern scientific community has largely accepted that open science requires sharing code. When publishing a paper, authors frequently include a link to a GitHub repository in the footnotes, assuming this satisfies the requirement for transparency.
However, as Tatjana Bolić et al. argue in Roadmap for a European Open Science Alliance for ATM Research, many technical publications still amount to "scientific advertising." The text describes a brilliant result, but the actual computational pipeline remains a black box. The core issue is a widening reproducibility gap: the distance between a static PDF with a code link and a truly verifiable, executable experiment.
To build reliable, cumulative knowledge, we must understand why simple code sharing fails and how the field is moving toward full artifact verification.
The Illusion of the GitHub Link
Why does a repository link fail to guarantee reproducibility? The answer lies in the hidden complexity of execution context.
Mohammad Arvan et al. investigated this directly in Reproducibility in Computational Linguistics: Is Source Code Enough?. Despite the presence of source code, the authors encountered a dismal 25% success rate when attempting to reproduce papers from a major NLP conference. They identified several systemic points of failure:
- Dependency Hell: Code relies on specific versions of external libraries. Without exact environment pinning, subsequent updates to those libraries break the original code.
- Missing Orchestration Logic: Repositories often contain raw source code but lack the execution scripts, command-line arguments, or data processing pipelines required to regenerate the exact metrics reported in the paper.
- Undocumented Hardware Requirements: Algorithms (especially deep learning models) may behave differently or fail entirely without specific GPU configurations or memory thresholds.
Furthermore, code repositories are dynamic, while papers are static. Frank Schindler et al. note in Author Correction: Higher-order topology in bismuth that general repository links often lack the granularity required for complex computational experiments. Effective open science requires precise pointers to specific sub-directories or immutable code versions. When links break or point to shifting codebases, the scientific foundation crumbles, requiring formal errata simply to restore access (Eaman Jahani et al., Erratum to: Improving official statistics in emerging markets using machine learning and mobile phone data).
The Hidden Components of the Pipeline
An executable script is still useless without its corresponding data environment. A computational experiment is a delicate triad of code, data, and environment. If any component is missing or unversioned, the pipeline breaks.
[ Raw Data ] ------> ( Hidden Preprocessing ) ------> [ Data Splits / Indexes ]
|
v
[ Undocumented Environment ] ---> ( Missing Orchestration ) ---> [ Final Metrics ]
In information retrieval, for example, Odunayo Ogundepo et al. point out in Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face that the specific data indexes used in experiments are rarely shared in a versioned format. Without access to the exact index, external researchers cannot precisely replicate retrieval performance.
Similarly, Daniel Garijo demonstrates in Semantic Workflows and Machine Learning for the Assessment of Carbon Storage by Urban Trees how undocumented manual processing steps and opaque custom scripts create black boxes in environmental modeling. As Fernando Chirigati et al. summarize in A Collaborative Approach to Computational Reproducibility, presenting results as static text and images without providing the underlying executable environment or data processing scripts makes independent validation nearly impossible.
The Cost of Opacity
When computational pipelines remain opaque, the scientific community suffers from an "evaluation crisis" (Chao Liu et al., Opportunities and Challenges in Code Search Tools). Without a transparent pipeline, it is impossible to distinguish between a genuine algorithmic breakthrough and an advantage gained from hyperparameter tuning or undisclosed data configurations.
A stark example of this occurred in the hardware design space. Chung-Kuan Cheng et al. published an independent evaluation in Assessment of Reinforcement Learning for Macro Placement targeting a high-profile reinforcement learning (RL) method for chip design. The authors discovered that the RL method's claimed success relied heavily on specific, undisclosed initial placement data. When evaluated transparently against standardized baselines, the complex RL method often underperformed simpler, traditional techniques.
Engineering the Solution: From Code to Artifacts
To bridge the reproducibility gap, the research community is shifting its focus from publishing static documents to engineering verifiable, holistic artifacts. This movement operates on two fronts: institutional badging and continuous workflow tooling.
Institutional Badging and Artifact Review
Major venues are changing their review structures to incentivize transparency. Journals like PVLDB and PoPETs have introduced formal artifact review tracks (Fatma Ozcan et al., Front Matter (Proc. VLDB Endow. 2022); Aaron Johnson et al., Editors' Introduction (Proc. Priv. Enhancing Technol. 2021)). Dedicated reviewers test the authors' code, datasets, and proofs to ensure they run and yield the reported results, awarding "Artifacts Available" or "Artifacts Reproduced" badges.
Continuous Workflow Tooling
To prevent reproducibility from being an afterthought, new tools aim to capture the entire lifecycle of an experiment:
- Research Objects: Jun Zhao 0003 et al. introduce RO-Manager in RO-Manager: A Tool for Creating and Manipulating Research Objects to Support Reproducibility and Reuse in Sciences to help scientists package materials, metadata, and methods into a unified "Research Object" from the early stages of the research cycle.
- Living Documents: To eliminate the disconnect between dynamic code and static narratives, Daniel Himmelstein et al. propose Manubot in Open collaborative writing with Manubot. Manubot adopts the "Git workflow" of software development for scientific writing, automatically tracking data provenance and updating the manuscript when underlying code or data changes.
True computational reproducibility requires treating the entire pipeline—the code, the data splits, the environment configurations, and the orchestration logic—as a first-class citizen of the scientific record. Only by moving beyond simple code dumps can the community ensure that published breakthroughs are reliable foundations for future innovation.
Key Foundational Work
Exploring the Coming Repositories of Reproducible Experiments: Challenges and Opportunities (2011) — contributed 1 concept elements to this direction.
Navigating large-scale semi-structured data in business portals (2001) — contributed 1 concept elements to this direction.