Coverage and Cynicism: The AFRL Submission to the WMT 2018 Parallel Corpus Filtering Task

Grant Erdmann, Jeremy GwinnupEMNLP 2018

TL;DR

This paper presents AFRL's approach to the WMT 2018 Parallel Corpus Filtering Task, introducing two novel metrics (Coverage and Cynical) for selecting high-quality parallel data for machine translation training. Their best-performing submission, incorporating a translation score, significantly outperformed the average of other participants, especially for smaller training sets.

What Problem Does This Paper Address?

Training effective machine translation (MT) systems, particularly neural MT (NMT), requires large amounts of high-quality parallel text (sentences aligned in two languages). However, real-world parallel corpora are often noisy, containing misalignments, non-parallel sentences, and linguistic errors. This noise can degrade MT performance. The WMT 2018 Parallel Corpus Filtering Task specifically aimed to evaluate methods for filtering these noisy corpora to create cleaner, more useful training sets. AFRL's work addresses the challenge of efficiently and effectively identifying and selecting the most valuable sentence pairs from a massive, noisy dataset.

Methods

AFRL developed two primary metrics for evaluating sentence pair quality:

  • Coverage Metric: This metric aims to maximize the similarity of a selected subset of parallel sentences to a representative target set. It uses a submodular function (specifically, log(1+x)) to quantify coverage and includes an "oversaturation penalty" to discourage redundant content.
  • Cynical Metric: Based on Axelrod's (2017) work, this metric seeks to minimize cross-entropy between language models built from the selected parallel data. AFRL extended this to include n-grams (up to 4-grams), deviating from the original unigram-only approach.

Both metrics are used within a greedy set-building algorithm that iteratively adds the line with the best metric improvement and then checks for potential removals to further enhance the score.

Additionally, AFRL employed a multi-stage preprocessing pipeline:

  1. Rough Preprocessing: Filters out lines based on length, presence of "www," word ratio between languages, and non-standard Unicode characters.
  2. Text Normalization: Includes punctuation normalization, word truncation, and language-specific tokenization using Byte-Pair Encoding (BPE).
  3. Translation Score: A German-English NMT system (OpenNMT) was used to translate German sentences, and their Meteor scores against the original English were computed. This translation score was then incorporated into the coverage metric (cvg-mix-meteor).

Datasets

The primary dataset used was the WMT 2018 news translation task parallel corpus, containing approximately 10^8 lines of English-German data. The Paracrawl corpus was also used. For training their internal NMT system, they excluded the Paracrawl corpus. Evaluation was performed on various test sets, including newstest2017, newstest2018, iwslt2017, Acquis, EMEA, Global, Ves, and KDE.

Key Insights and Ideas

  • Translation Score is Crucial: Incorporating a translation score (like Meteor) into the filtering process proved highly beneficial, especially for smaller training sets, leading to their best-performing submission.
  • Cynical Metric for NMT: The Cynical metric showed a significant advantage over the Coverage metric for NMT systems trained on small datasets, suggesting its effectiveness in capturing subtle linguistic nuances relevant to NMT.
  • Preprocessing Trade-offs: The aggressive initial filtering (removing up to 90% of data) might have removed beneficial lines. The authors suggest that applying the MT system before filtering could be more optimal, though computationally more expensive.
  • Set Size Matters: The ability to specify the desired size of the selected set was beneficial for the Coverage metric, allowing for tailored training data sizes.

Key Results

AFRL submitted five systems to the WMT 2018 Parallel Filtering Task. Their best system, cvg-mix-meteor, which combined the coverage metric with a translation score, achieved the highest mean BLEU score across all MT systems and test sets for the 106-word SMT task. This system also performed competitively on smaller training sets (10^6 words) for both SMT and NMT, outperforming the median and many other participants. The Cynical metric was particularly effective for NMT, yielding better results than the Coverage metric for small training sets.

Limitations and Future Work

  • Aggressive Pre-filtering: The paper acknowledges that their initial filtering steps might have removed valuable data, and suggests that applying the translation scoring earlier in the pipeline could be more effective, albeit computationally intensive.
  • Cynical Metric Set Size: Specifying a desired output set size for the Cynical metric is less straightforward than for the Coverage metric.
  • Computational Cost: While the Cynical metric is computationally less burdensome than Coverage, the overall process, especially incorporating translation scores, can be resource-intensive. Future work could focus on optimizing these processes and exploring more efficient ways to integrate translation quality into filtering.

Summary updated

BibTeX
@inproceedings{erdmann2018coverage,
  author = {Grant Erdmann and Jeremy Gwinnup},
  title = {{Coverage and Cynicism: The AFRL Submission to the WMT 2018 Parallel Corpus Filtering Task}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure