Beyond Code Sharing for Computational Reproducibility

53 concepts36 papers1 proposal

The modern scientific community has largely accepted that open science requires sharing code. When publishing a paper, authors frequently include a link to a GitHub repository in the footnotes, assuming this satisfies the requirement for transparency.

However, as Tatjana Bolić et al. argue in Roadmap for a European Open Science Alliance for ATM Research, many technical publications still amount to "scientific advertising." The text describes a brilliant result, but the actual computational pipeline remains a black box. The core issue is a widening reproducibility gap: the distance between a static PDF with a code link and a truly verifiable, executable experiment.

To build reliable, cumulative knowledge, we must understand why simple code sharing fails and how the field is moving toward full artifact verification.

Why does a repository link fail to guarantee reproducibility? The answer lies in the hidden complexity of execution context.

Mohammad Arvan et al. investigated this directly in Reproducibility in Computational Linguistics: Is Source Code Enough?. Despite the presence of source code, the authors encountered a dismal 25% success rate when attempting to reproduce papers from a major NLP conference. They identified several systemic points of failure:

  • Dependency Hell: Code relies on specific versions of external libraries. Without exact environment pinning, subsequent updates to those libraries break the original code.
  • Missing Orchestration Logic: Repositories often contain raw source code but lack the execution scripts, command-line arguments, or data processing pipelines required to regenerate the exact metrics reported in the paper.
  • Undocumented Hardware Requirements: Algorithms (especially deep learning models) may behave differently or fail entirely without specific GPU configurations or memory thresholds.

Furthermore, code repositories are dynamic, while papers are static. Frank Schindler et al. note in Author Correction: Higher-order topology in bismuth that general repository links often lack the granularity required for complex computational experiments. Effective open science requires precise pointers to specific sub-directories or immutable code versions. When links break or point to shifting codebases, the scientific foundation crumbles, requiring formal errata simply to restore access (Eaman Jahani et al., Erratum to: Improving official statistics in emerging markets using machine learning and mobile phone data).

The Hidden Components of the Pipeline

An executable script is still useless without its corresponding data environment. A computational experiment is a delicate triad of code, data, and environment. If any component is missing or unversioned, the pipeline breaks.

   [ Raw Data ] ------> ( Hidden Preprocessing ) ------> [ Data Splits / Indexes ]
                                                                   |
                                                                   v
[ Undocumented Environment ] ---> ( Missing Orchestration ) ---> [ Final Metrics ]

In information retrieval, for example, Odunayo Ogundepo et al. point out in Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face that the specific data indexes used in experiments are rarely shared in a versioned format. Without access to the exact index, external researchers cannot precisely replicate retrieval performance.

Similarly, Daniel Garijo demonstrates in Semantic Workflows and Machine Learning for the Assessment of Carbon Storage by Urban Trees how undocumented manual processing steps and opaque custom scripts create black boxes in environmental modeling. As Fernando Chirigati et al. summarize in A Collaborative Approach to Computational Reproducibility, presenting results as static text and images without providing the underlying executable environment or data processing scripts makes independent validation nearly impossible.

The Cost of Opacity

When computational pipelines remain opaque, the scientific community suffers from an "evaluation crisis" (Chao Liu et al., Opportunities and Challenges in Code Search Tools). Without a transparent pipeline, it is impossible to distinguish between a genuine algorithmic breakthrough and an advantage gained from hyperparameter tuning or undisclosed data configurations.

A stark example of this occurred in the hardware design space. Chung-Kuan Cheng et al. published an independent evaluation in Assessment of Reinforcement Learning for Macro Placement targeting a high-profile reinforcement learning (RL) method for chip design. The authors discovered that the RL method's claimed success relied heavily on specific, undisclosed initial placement data. When evaluated transparently against standardized baselines, the complex RL method often underperformed simpler, traditional techniques.

Engineering the Solution: From Code to Artifacts

To bridge the reproducibility gap, the research community is shifting its focus from publishing static documents to engineering verifiable, holistic artifacts. This movement operates on two fronts: institutional badging and continuous workflow tooling.

Institutional Badging and Artifact Review

Major venues are changing their review structures to incentivize transparency. Journals like PVLDB and PoPETs have introduced formal artifact review tracks (Fatma Ozcan et al., Front Matter (Proc. VLDB Endow. 2022); Aaron Johnson et al., Editors' Introduction (Proc. Priv. Enhancing Technol. 2021)). Dedicated reviewers test the authors' code, datasets, and proofs to ensure they run and yield the reported results, awarding "Artifacts Available" or "Artifacts Reproduced" badges.

Continuous Workflow Tooling

To prevent reproducibility from being an afterthought, new tools aim to capture the entire lifecycle of an experiment:

True computational reproducibility requires treating the entire pipeline—the code, the data splits, the environment configurations, and the orchestration logic—as a first-class citizen of the scientific record. Only by moving beyond simple code dumps can the community ensure that published breakthroughs are reliable foundations for future innovation.

Key Foundational Work

Exploring the Coming Repositories of Reproducible Experiments: Challenges and Opportunities (2011) — contributed 1 concept elements to this direction.
Navigating large-scale semi-structured data in business portals (2001) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reproducibility_gap_a7e636af,
  title = {reproducibility gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reproducibility-gap/rdr_a7e636af70d645e289662c9751d0a2da},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_reproducibility_gap_a7e636af_v1,
  title = {reproducibility gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reproducibility-gap/rdr_a7e636af70d645e289662c9751d0a2da/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake