On the Replicability and Reproducibility of Deep Learning in Software Engineering

Chao Liu, Cuiyun Gao, Xin Xia, David Lo, John C. Grundy, Xiaohu Yang

Abstract

Deep learning (DL) has become a cornerstone of modern software engineering (SE) research, powering breakthroughs in code search, bug detection, and automated documentation. However, the stochastic nature of these models—relying on random initialization and non-deterministic optimization—raises critical questions about the stability and generalizability of reported results. This study audits 93 recent DL-based SE papers and performs empirical re-runs of four representative models to quantify the "credibility gap" in the field.

Introduction: The Credibility Gap

In scientific research, two pillars ensure the validity of findings: replicability (obtaining the same results with the same model and data) and reproducibility (obtaining consistent results with the same protocol on different data). In the SE community, DL models are frequently treated as "black boxes," where researchers report peak performance from a single "lucky" run without accounting for inherent variance. This paper investigates how prevalent this lack of transparency is and how it affects the perceived success of state-of-the-art models.

A Meta-Analysis of the SE Literature

The authors conducted a comprehensive literature review of 93 DL-based SE papers published between 2015 and 2019. The selection targeted top-tier venues, reflecting the highest standards of the community.

Breakdown of surveyed papers by year and publication type (2015-2019).

As shown in the table above, the volume of DL research in SE grew significantly over the five-year period, with the vast majority of papers appearing in conferences. The distribution across specific venues highlights the broad interest in DL across the SE spectrum.

Distribution of the 93 surveyed papers across Software Engineering venues.

The audit revealed a troubling transparency crisis: only 25.8% of these papers provided accessible replication packages, and a mere 10.8% discussed research questions related to the stability or reproducibility of their findings.

Empirical Investigation: Re-running the Models

To move beyond a literature audit, the authors re-evaluated four representative DL models:

  • DeepCS: Code search via information retrieval.
  • RRGen: App review response generation using RNNs.
  • RLNN: PR description generation via reinforcement learning.
  • ASTNN: Code classification using Abstract Syntax Trees (ASTs).

These models were tested against four critical factors: model stability (weight randomness), convergence (training duration), vocabulary sensitivity (Out-of-Vocabulary issues), and testing data scale.

Key Insights

  1. The "Best-Case" Bias: Reported results for two models were inflated by over 12% compared to the average of ten runs. This suggests that "state-of-the-art" claims may often be based on statistical outliers rather than representative performance.
  2. Sensitivity to Vocabulary: For the DeepCS model, simply increasing the fixed vocabulary size led to a performance boost of 104.2%. This indicates that architectural superiority is often conflated with data preprocessing choices.
  3. Structural Robustness: The ASTNN model, which leverages the structural logic of code (ASTs) rather than treating it as plain text, proved to be the most stable and reproducible. This suggests that domain-specific architectures are more resilient to stochastic fluctuations.
  4. Convergence and Data Scale: Models that have not fully converged exhibit "turbulent" performance, with swings of ±8%. Furthermore, results on small benchmarks often fail to scale to larger, real-world datasets.

Conclusion and Recommendations

The study concludes that the current culture of reporting single-run "best" results in SE research is misleading. To bridge the credibility gap, the authors propose several mandates for the community:

  • Mandatory Replication Packages: Journals and conferences should require long-lasting, public links to code and data.
  • Statistical Reporting: Researchers should report the mean and standard deviation across multiple runs to provide a realistic view of model performance.
  • Focus on Structural Models: The stability of AST-based models suggests that incorporating SE-specific knowledge into DL architectures is a promising path toward more reproducible research.

By addressing these issues, the SE community can ensure that the "deep learning revolution" is built on a foundation of scientific rigor rather than lucky seeds.

Summary updated

Abstract

Deep learning (DL) techniques have gained significant popularity among software engineering (SE) researchers in recent years. This is because they can often solve many SE challenges without enormous manual feature engineering effort and complex domain knowledge. Although many DL studies have reported substantial advantages over other state-of-the-art models on effectiveness, they often ignore two factors: (1) replicability - whether the reported experimental result can be approximately reproduced in high probability with the same DL model and the same data; and (2) reproducibility - whether one reported experimental findings can be reproduced by new experiments with the same experimental protocol and DL model, but different sampled real-world data. Unlike traditional machine learning (ML) models, DL studies commonly overlook these two factors and declare them as minor threats or leave them for future work. This is mainly due to high model complexity with many manually set parameters and the time-consuming optimization process. In this study, we conducted a literature review on 93 DL studies recently published in twenty SE journals or conferences. Our statistics show the urgency of investigating these two factors in SE. Moreover, we re-ran four representative DL models in SE. Experimental results show the importance of replicability and reproducibility, where the reported performance of a DL model could not be replicated for an unstable optimization process. Reproducibility could be substantially compromised if the model training is not convergent, or if performance is sensitive to the size of vocabulary and testing data. It is therefore urgent for the SE community to provide a long-lasting link to a replication package, enhance DL-based solution stability and convergence, and avoid performance sensitivity on different sampled data.
BibTeX
@article{liu2022replicability,
  author = {Chao Liu and Cuiyun Gao and Xin Xia and David Lo and John C. Grundy and Xiaohu Yang},
  title = {{On the Replicability and Reproducibility of Deep Learning in Software Engineering}},
  journal = {ACM Trans. Softw. Eng. Methodol.},
  year = {2022},
  doi = {10.1145/3477535},
}
Figure