When a researcher publishes source code on GitHub, it feels like a permanent contribution to the scientific record. The code is frozen in time, a static sequence of characters that perfectly captures the logic of an experiment. Yet, if you try to run that exact same code a few years—or sometimes even a few months—later, there is a high probability it will fail.
This failure is not caused by a change in the code itself, but by a shift in the world around it. This phenomenon is known as software bit rot or dependency decay. To understand why it happens, we have to look at code not as an isolated artifact, but as the top layer of a massive, constantly shifting software stack.
The Illusion of the Source Code Artifact
The modern scientific workflow relies heavily on open-source sharing. However, simply providing a link to a repository does not guarantee long-term reproducibility. In a hands-on study of computational linguistics papers, Arvan et al. in Reproducibility in Computational Linguistics: Is Source Code Enough? found a dismal 25% success rate in reproducing results, largely thwarted by "dependency hell" and undocumented environments.
When code fails to run outside its original context, it exposes what Rougier et al. in Sustainable computational science: the ReScience initiative call the "works on my machine" phenomenon. The code itself is merely a recipe; it requires an entire kitchen of specific tools, ingredients, and appliances to execute. If any of those tools change, the recipe breaks.
The Transitive Closure of Execution
To understand why software decays, we must consider what Satyanarayanan et al. describe in Collaborating with executable content across space and time as the transitive closure of an execution environment. A single script does not run in a vacuum. It relies on a deep hierarchy of dependencies:
$$\text{Code} \longrightarrow \text{Libraries} \longrightarrow \text{Language Runtime/Compiler} \longrightarrow \text{Operating System} \longrightarrow \text{Hardware Drivers}$$
If any single component within this transitive closure updates, deprecates, or disappears, the entire stack collapses. This makes computational research incredibly brittle. As software ecosystems evolve, the environment that once supported an experiment dissolves beneath it.
The Three Mechanisms of Decay
Software decay typically manifests through three distinct mechanisms: API evolution, compiler strictness, and silent behavioral drift.
1. API Evolution and Deprecation
Third-party libraries are living codebases. Functions are renamed, arguments are restructured, and old interfaces are deleted to make room for optimization. Fischer et al. highlight this in mlr3extralearners: Expanding the mlr3 Ecosystem with Community-Driven Learner Integration, noting that maintaining integrations across hundreds of external packages requires continuous vigilance because upstream updates routinely break downstream wrappers.
This problem even impacts artificial intelligence. Pesaran Zadeh et al. observe in Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedback that code generation models trained on specific library versions quickly become obsolete as the underlying APIs undergo updates, requiring constant maintenance to keep the generated code functional.
2. Compiler Strictness
Even if your library dependencies remain unchanged, the tools used to compile or interpret the code evolve. Over time, compilers become stricter to enforce better security and performance standards. Bonneel et al., in their extensive study Code Replicability in Computer Graphics, discovered that a primary barrier to running older graphics code was that newer compilers simply refused to build it due to breaking changes in compiler strictness and language standards.
3. Silent Behavioral Drift
The most insidious form of decay occurs when the code still runs, but the results change. Kosiński notes in archivist: An R Package for Managing, Recording and Restoring Data Analysis Results that updates to underlying mathematical or statistical libraries can cause identical scripts to yield different outputs over time. Furthermore, Popovic et al. demonstrate in Exploring Variation of Results from Different Experimental Conditions that minor variations in runtime environments, hardware architectures, or library versions can swing experimental performance scores significantly, muddying scientific verification.
The Temporal Dimension of Rigor
This reality forces us to re-evaluate what it means for research to be "rigorous." Traditionally, peer review treats reproducibility as a static checkmark at the time of publication. However, Raff et al. argue in What Do Machine Learning Researchers Mean by "Reproducible"? that scientific rigor possesses an overlooked temporal dimension. Because technical environments shift, maintaining the validity of a finding requires an ongoing, iterative process of replication across time.
Without active maintenance, interactive components and live demonstrations inevitably break down, a challenge documented by Nicolaï et al. in The approach to reproducible research of the Image Processing On Line (IPOL) journal. Similarly, Do et al. show in The Role of the Adversary Model in Applied Security Research that failing to precisely document environment constraints and operating system versions quickly renders historical security evaluations ambiguous and unrecoverable.
Preserving the Computational Ecosystem
If source code alone cannot withstand time, how do we preserve digital scholarship? The consensus across recent literature points away from sharing raw code and toward preserving entire computational ecosystems.
[Traditional Sharing] Source Code Only ──> (Environment Shifts) ──> Execution Fails
[Modern Preservation] Source Code + Containerized Environment ──> Execution Succeeds
As Chirigati et al. explain in A Collaborative Approach to Computational Reproducibility, ensuring that experiments remain executable decades later requires capturing the exact state of the underlying software environment. Even when using virtualization, Freire et al. note in Provenance and the Different Flavors of Reproducibility that tracking complete digital "paper trails" or provenance data remains an ongoing challenge for long-term archival.
To combat this, modern frameworks are shifting toward strict environment isolation. For instance, in rapidly moving fields like computer vision, Kulhanek et al. introduced NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods, which uses isolated software environments to prevent dependency rot from compromising benchmark leaderboards.
Ultimately, solving bit rot requires a cultural shift in how science treats software. As originally argued by Prlić et al. in The PLOS Computational Biology Software Section, software must be treated as a first-class research contribution tied to stable, permanent repositories. Initiatives like the NeurIPS reproducibility program, detailed by Pineau et al. in Improving Reproducibility in Machine Learning Research, demonstrate that mandatory code checklists and environment documentation significantly improve the initial survival rate of research artifacts. However, keeping those artifacts alive across decades demands that we look beyond the script and actively preserve the fragile digital ecosystems that allow them to run.
Key Foundational Work
The PLOS Computational Biology Software Section (2012) — contributed 1 concept elements to this direction.
Book Review: Machine Learning: An Algorithmic Perspective (2009) — contributed 1 concept elements to this direction.