When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP

Sara Papi, Marco Gaido, Andrea Pilzer, Matteo Negri

Introduction

In modern Natural Language Processing (NLP), reproducibility has become the gold standard for scientific validity. The community has responded to the "reproducibility crisis" by implementing checklists and requiring open-source code releases. However, this paper argues that we have overlooked a critical distinction: reproducibility is not correctness.

If a researcher releases code that allows others to achieve the same high scores, the work is currently deemed "sound." Yet, if those scores are achieved due to underlying implementation errors, the research becomes a "giant with feet of clay"—an unstable foundation that misguides future work. This study investigates the lack of Software Quality Assurance (SQA) in NLP and demonstrates how "silent bugs" in state-of-the-art architectures can lead to fundamentally flawed scientific conclusions.

The Correctness Gap in NLP Research

To understand the scope of the problem, the authors conducted a conference audit of top-tier venues including ACL, NeurIPS, and ICLR. They reviewed submission forms to determine if "code correctness" is explicitly evaluated. The results indicate that while reproducibility is increasingly prioritized, the actual functional correctness of the underlying code is largely neglected.

An audit of major NLP and AI venues showing that while reproducibility is often checked, code correctness is rarely a formal requirement.

Methodology: Stress-Testing for Silent Bugs

The authors focused their investigation on the Conformer architecture, widely used in speech-to-text tasks. They employed an "Inference Batch Size" (IBS) stress test: in a correct implementation, results should remain identical regardless of batch size. Discrepancies indicate that padding (added to normalize sequence lengths in a batch) is being incorrectly processed as actual data.

Through this method, they identified three specific bug categories:

  1. Convolutional Bugs: Modules failing to ignore padding during depthwise or pointwise convolutions.
  2. Subsampling Bugs: Initial layers leaking non-zero values from padded areas into the computation.
  3. Positional Encoding Bugs: Relative positional encodings shifting incorrectly in the presence of padding.

Widespread Software Quality Issues

The audit of six popular open-source repositories—including Fairseq, ESPnet, and NeMo—revealed that these issues are nearly universal. Every repository analyzed contained at least one of the identified bugs in their Conformer implementations.

A summary of identified bugs across major open-source speech processing repositories, highlighting the prevalence of implementation errors.

Key Insights and Results

The study's most startling finding is that these bugs are often "silent" or even "beneficial" at small batch sizes. In some instances, buggy code outperformed correct code because the model learned to exploit information leaked through incorrect padding handling.

However, this performance is fragile. When the Inference Batch Size was increased to 100, the Word Error Rate (WER) on ASR tasks using the MuST-C v1.0 dataset skyrocketed from 10.52 (correct) to 54.56 (buggy). Furthermore, the authors demonstrated that building on buggy code can lead to false breakthroughs; a technique called "CTC compression" appeared to provide a massive, statistically significant boost on buggy code that vanished when tested on a correct implementation.

Conclusion and the PangoliNN Solution

To address these systemic issues, the authors introduced pangoliNN, a Python library designed for unit-testing neural modules. They also proposed a new Code-quality Checklist for the research community. While some fixes (particularly for positional encoding) introduce computational overhead, the authors argue that the cost of correctness is necessary to ensure the scientific integrity of the field. They urge conference organizers to integrate SQA into the official review process to move beyond simple portability and toward true functional correctness.

Summary updated

Abstract

Despite its crucial role in research experiments, code correctness is often presumed solely based on the perceived quality of results. This assumption, however, comes with the risk of erroneous outcomes and, in turn, potentially misleading findings. To mitigate this risk, we posit that the current focus on reproducibility should go hand in hand with the emphasis on software quality. We support our arguments with a case study in which we identify and fix three bugs in widely used implementations of the state-of-the-art Conformer architecture. Through experiments on speech recognition and translation in various languages, we demonstrate that the presence of bugs does not prevent the achievement of good and reproducible results, which however can lead to incorrect conclusions that potentially misguide future research. As countermeasures, we release pangoliNN, a library dedicated to testing neural models, and propose a Code-quality Checklist, with the goal of promoting coding best practices and improving software quality within the NLP community.
BibTeX
@inproceedings{papi2024when,
  author = {Sara Papi and Marco Gaido and Andrea Pilzer and Matteo Negri},
  title = {{When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.200},
}
Figure