When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP

Sara Papi, Marco Gaido, Andrea Pilzer, Matteo Negri

TL;DR

Just because a machine learning model achieves state-of-the-art results and is perfectly reproducible doesn't mean the code is correct. This paper reveals that widely used open-source implementations of the popular Conformer architecture contain critical bugs in how they handle padding. While these bugs don't prevent the models from achieving excellent results, they can completely distort scientific conclusions when testing new techniques, highlighting a urgent need for software quality assurance in AI research.

What Problem Does This Paper Address?

In recent years, the machine learning and natural language processing (NLP) communities have fiercely combated the "reproducibility crisis" by introducing checklists and requiring open-source code repositories. However, the authors point out a dangerous blind spot: the community frequently conflates reproducibility and good performance with code correctness.

If a paper provides code that consistently reproduces a top-tier score, reviewers and subsequent researchers assume the code accurately executes its intended mathematical design. As the authors show, this assumption is a "giant with feet of clay." Buggy code can still yield excellent results, but it acts as a corrupted foundation for future research, leading scientists to draw completely false conclusions about the efficacy of new methods.

Review forms from major NLP venues show a stark neglect of code correctness compared to reproducibility.

Methods

The authors audited six widely used open-source repositories implementing the Conformer architecture—the current state-of-the-art for speech processing—including Fairseq-ST, ESPnet-ST, NeMo, SpeechBrain, TorchAudio, and a popular standalone repository.

To detect hidden flaws, they systematically varied a parameter that mathematically should not alter a model's final output: the Inference Batch Size (IBS). When processing batches of varying sequence lengths, frameworks append zero-value "padding" tokens to align the sequences. In a correct implementation, padding should be completely ignored and have zero impact on the final predictions.

Through this audit, they uncovered three distinct padding-related bugs across the repositories:

  1. Conformer Convolutions Bug: Depthwise and pointwise convolutions fail to account for padding, generating non-zero values in padded zones that corrupt subsequent batch normalization layers.
  2. Initial Subsampling Bug: The initial downsampling convolutions process padding tokens as valid data, corrupting the edges of the true sequence.
  3. Positional Encodings Bug: The relative shift operation applied to the positional encoding matrix incorrectly shifts padding values into the active, non-padded matrix area.
Example of the relative shift operation with a bug (2) vs. a correct implementation (3).

To help the community prevent these issues, the authors developed and released pangoliNN, a PyTorch-based unit testing library designed to automatically verify core neural network properties like padding invariance and causality. They also proposed a 5-question Code-quality Checklist for machine learning conferences.

Datasets

The authors evaluated their findings on the MuST-C v1.0 corpus, a massive multilingual speech-to-text translation and automatic speech recognition (ASR) benchmark.

  • For ASR, they utilized the English-Spanish (en-es) section.
  • For Speech Translation (ST), they evaluated models across eight language pairs, translating English audio into Dutch (nl), French (fr), German (de), Italian (it), Portuguese (pt), Romanian (ro), Russian (ru), and Spanish (es).

Key Insights and Ideas

  • Bugs Can Hide Behind SOTA Results: Even with all three padding bugs active, the Conformer models still comfortably outperformed standard Transformer baselines and matched published state-of-the-art results. Because the performance was excellent, these bugs went unnoticed by thousands of researchers.
  • Flawed Code Distorts the Scientific Method: The most alarming finding occurred when the authors tested a popular technique called CTC compression on the buggy code. On the broken codebase, CTC compression appeared to deliver massive, statistically significant performance gains. However, when they ran the exact same experiment on their corrected codebase, the gains completely vanished. CTC compression wasn't actually improving speech processing; it was simply shortening the sequences, which accidentally reduced the amount of padding and masked the underlying bugs!
Distribution of the three identified padding bugs across six major open-source repositories.

Key Results

  • The Impact of Batching: On the buggy codebase, increasing the Inference Batch Size (IBS) from 1 to 100 caused the Word Error Rate (WER) in ASR to skyrocket from 11.32% to 54.56% due to the accumulation of padding errors. On the corrected codebase, the WER remained perfectly stable at 10.52% regardless of batch size.
  • The Illusion of Progress: When evaluating speech translation across all 8 languages, running CTC compression on the buggy codebase yielded an average artificial boost of up to 4.1 BLEU points at a batch size of 100. On the bug-free codebase, the actual average improvement was a negligible 0.29 BLEU points.

Limitations and Future Work

While the authors successfully resolved all three bugs, they note that their fix for the Positional Encodings Bug introduces a significant computational overhead, which increases model training times. They open-sourced their solution in hopes that the community will contribute optimizations to eliminate this bottleneck. Additionally, this study focused strictly on speech recognition and translation using the Conformer architecture; future work is required to audit other popular architectures and domains (such as text-to-speech or speech separation) for similar hidden software quality flaws.

Summary updated

Abstract

Despite its crucial role in research experiments, code correctness is often presumed only on the basis of the perceived quality of results. This assumption comes with the risk of erroneous outcomes and potentially misleading findings. To address this issue, we posit that the current focus on reproducibility should go hand in hand with the emphasis on software quality. We present a case study in which we identify and fix three bugs in widely used implementations of the state-of-the-art Conformer architecture. Through experiments on speech recognition and translation in various languages, we demonstrate that the presence of bugs does not prevent the achievement of good and reproducible results, which however can lead to incorrect conclusions that potentially misguide future research. As a countermeasure, we propose a Code-quality Checklist and release pangoliNN, a library dedicated to testing neural models, with the goal of promoting coding best practices and improving research software quality within the NLP community.
BibTeX
@inproceedings{papi2024when,
  author = {Sara Papi and Marco Gaido and Andrea Pilzer and Matteo Negri},
  title = {{When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.200},
}
Figure