Detecting Various Types of Noise for Neural Machine Translation

Christian Herold, Jan Rosendahl, Joris Vanvinckenroye, Hermann NeyACL 2022 · DOI

Introduction

Neural Machine Translation (NMT) models are notoriously sensitive to the quality of their training data. While the research community has developed various data filtering tools, their specific effectiveness against different types of noise has remained largely unquantified. This paper benchmarks state-of-the-art filtering systems—including Cross-Entropy (CE) scoring, LASER embeddings, and Language ID—against a variety of synthetic noise categories. The goal is to move beyond aggregate BLEU scores and understand exactly which types of "garbage" are being collected and which are slipping through the cracks.

Methodology

The study evaluates filtering performance across two language pairs representing different resource levels: high-resource German→English (De→En) and low-resource Khmer→English (Km→En). To establish a ground truth, the authors created synthetic noise categories by corrupting clean data. These categories include:

  • Misaligned Sentences: Shuffling target sentences so they no longer match the source.
  • Wrong Language/Untranslated: Inserting text from a different language or copying the source directly to the target.
  • Over/Undertranslation: Deleting portions of a sentence to simulate incomplete translations.
  • Synthetic Translations: Replacing human translations with machine-generated text.

Analysis of Noise Detection

High-Resource Performance (De→En)

In high-resource settings, current filtering pipelines are highly effective at identifying "obvious" noise. For De→En, detecting the wrong language or untranslated segments is nearly perfect, with accuracies reaching 97% when using a combination of Cross-Entropy and Language ID. However, more subtle and damaging errors are harder to spot. Misaligned sentences and overtranslations typically see detection accuracies stuck between 65% and 70%, meaning a significant portion of this "poisonous" data remains in the training set.

Filtering accuracy for various noise categories in the De→En language pair.

The Low-Resource Challenge (Km→En)

Performance degrades significantly in low-resource scenarios like Khmer→English. A primary bottleneck is the unreliability of Language ID tools for such languages, which can negatively impact the entire filtering pipeline. For example, the accuracy of detecting misordered words on the source side drops from 89% in German to just 63% in Khmer. In some instances, adding a brittle Language ID filter actually reduced overall performance compared to using standalone methods.

Filtering accuracy for various noise categories in the Km→En language pair.

Key Insights

  • Misalignment is a Stealth Killer: Despite its negative impact on NMT quality, misaligned data is surprisingly difficult for current state-of-the-art filters to detect reliably.
  • The MT Identity Crisis: Current systems struggle to distinguish high-quality machine translation from human translation. As the web becomes increasingly saturated with synthetic content, this poses a major challenge for future data crawling and model training.
  • LASER vs. Cross-Entropy: LASER tends to be more robust as a standalone tool, whereas Cross-Entropy scoring often yields better results when high-quality language models are available to support it.

Conclusion and Future Work

While existing tools are excellent at removing blatant junk, they struggle with subtle linguistic errors and low-resource language pairs. Future research should focus on developing filtering methods that are less dependent on Language ID prerequisites and more capable of identifying nuanced errors like overtranslation and sentence misalignment. Moving toward adaptive, score-based thresholds rather than fixed-percentage filtering may also improve data quality in real-world web crawls.

Summary updated

BibTeX
@inproceedings{herold2022detecting,
  author = {Christian Herold and Jan Rosendahl and Joris Vanvinckenroye and Hermann Ney},
  title = {{Detecting Various Types of Noise for Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.200},
}
Figure