Refining an Almost Clean Translation Memory Helps Machine Translation

Shivendra Bhardwaj, David Alfonso-Hermelo, Philippe Langlais, Gabriel Bernier-Colborne, Cyril Goutte, Michel SimardAMTA 2022 · OpenReview

Abstract: Shivendra Bhardwa, David Alfonso-Hermelo, Philippe Langlais, Gabriel Bernier-Colborne, Cyril Goutte, Michel Simard. Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track).

2022.

Introduction

In the field of Neural Machine Translation (NMT), the prevailing wisdom often suggests that more data is better. While significant effort is typically spent cleaning noisy web-crawled corpora, professional Translation Memories (TMs) are generally treated as "gold standard" data. This paper challenges that assumption, demonstrating that even high-quality, professionally curated TMs contain enough noise to degrade MT performance. By applying sophisticated filtering, the researchers show that "less is more" when it comes to training data quality.

The Nature of Noise in Professional TMs

Despite being created by professionals, TMs accumulate noise over years of use. This noise typically falls into two categories: mechanical noise (extraction or alignment errors) and human-induced noise (typos, grammatical slips, or loose translations). Even in the proprietary corporate TM used for this study, approximately 7.7% of the data was flagged as potentially problematic.

Examples of noisy sentence pairs found in professional translation memories, including misalignments and junk text.

Methodology

The authors compared five distinct approaches to identify and filter out noisy sentence pairs:

  • Heuristics: 13 hand-crafted rules checking for length ratios, number matching, and character encoding.
  • Feature-based Classifiers: Traditional ML models (SVM and Random Forest) trained on heuristic scores.
  • TMOP: An unsupervised tool specifically designed for TM cleaning using 25 features.
  • Deep-Learning (bi-LSTM): A supervised siamese neural network that learns sentence representations.
  • LASER: An unsupervised approach using multilingual sentence embeddings to calculate similarity in a shared vector space.

Experimental Results

The study utilized a massive 139.5-million sentence pair TM (TM-XL), focusing on a 14-million pair subset (MT-TRAIN) for engine training. The impact of filtering was evaluated using XLM and ConvS2S architectures.

Surprisingly, the unsupervised LASER method performed exceptionally well, reaching 84% accuracy on a manual test set. The most effective strategy, however, was the "Intersection All" approach, which combined multiple filters. This method discarded nearly 60% of the training data, reducing the set from 14M to 5.8M pairs.

BLEU scores over training epochs for various filtering methods, showing that filtered subsets consistently outperform the full baseline.

As illustrated in the results, cleaning the TM led to a significant gain of +1.22 BLEU points for the XLM engine. Beyond translation scores, filtering drastically improved "vocabulary health," reducing the number of gibberish or unrepresentable tokens in the training data from 76,000 to as few as 450.

Conclusion and Limitations

This research proves that micro-cleaning high-quality data is a worthwhile investment for NMT. While deep learning models are superior at catching subtle translation choices, even unsupervised embedding-based methods like LASER offer powerful filtering capabilities.

Future work remains to be done on more distant language pairs and morphologically complex languages, as this study focused exclusively on English-French. Nonetheless, the results provide a clear roadmap for Language Service Providers to optimize their existing TMs for machine learning applications.

Summary updated

BibTeX
@inproceedings{bhardwaj2022refining,
  author = {Shivendra Bhardwaj and David Alfonso-Hermelo and Philippe Langlais and Gabriel Bernier-Colborne and Cyril Goutte and Michel Simard},
  title = {{Refining an Almost Clean Translation Memory Helps Machine Translation}},
  booktitle = {AMTA},
  year = {2022},
  url = {https://openreview.net/forum?id=GET9NET_q1A},
}
Figure