Quality and Coverage: The AFRL Submission to the WMT19 Parallel Corpus Filtering for Low-Resource Conditions Task

Grant Erdmann, Jeremy GwinnupACL 2019

TL;DR

The Air Force Research Laboratory (AFRL) tackled the WMT19 Parallel Corpus Filtering task by developing a novel subcorpus selection method that combines "coverage" (similarity to a target corpus) and "quality" (translation accuracy using a trained MT system). This approach significantly improved machine translation performance for low-resource language pairs (English-Nepali and English-Sinhala) compared to baselines.

What Problem Does This Paper Address?

Training effective machine translation (MT) systems, especially for low-resource languages, relies heavily on the quality and relevance of the parallel corpora used. However, publicly available parallel data is often noisy, containing misalignments, incorrect translations, or non-linguistic content. Filtering these noisy corpora to select only the most useful sentence pairs is crucial but challenging. Traditional methods often focus on simple heuristics or coverage, potentially overlooking the actual translation quality. This paper addresses the need for a more sophisticated filtering approach that balances corpus coverage with the practical utility of the data for training MT systems.

Methods

The AFRL submission introduces a two-pronged approach to subcorpus selection, integrating both "coverage" and "quality" metrics.

  • Coverage Metric: This metric, based on prior work (Gwinnup et al., 2016), aims to select a subcorpus that is similar to a representative target corpus. It uses a submodular function (specifically, a log-based function) to quantify the similarity of sentence pair counts within the subcorpus relative to the target corpus. A "preferred size" parameter (τ) is used to guide the selection towards a desired subcorpus size.
  • Text Quality Metric: This is a key innovation. Instead of relying solely on coverage, the authors train a preliminary MT system on the clean parallel data provided by the task organizers. This system is then used to translate the noisy parallel corpora. The quality of the given English sentences in the noisy corpus is then evaluated by comparing them to the translated English sentences using the Meteor metric. Meteor was chosen over BLEU for its deeper linguistic analysis.
  • Combined Score: The final score for a subcorpus is the product of its coverage score and its text quality score.
  • Subcorpus-Building Algorithm: A greedy algorithm iteratively adds the line that most improves the combined score. It also includes a step to remove lines if their removal increases the score, preventing infinite cycling.
  • Preprocessing: Before scoring, a rough filter removes lines exceeding 80 words (the limit for the test systems) and lines where language identification probabilities (from fastText) do not match the expected languages.

Datasets

  • Training Data: The task provided parallel corpora for English-Nepali (en-ne) and English-Sinhala (en-si). Both parallel and monolingual training datasets were available.
  • Evaluation Data: The filtering methods were evaluated by their impact on the performance of SMT and NMT systems trained on 1M and 5M English words of the filtered corpora.
  • Clean Data: A small amount of "clean" parallel data was provided by the organizers and used to train the initial MT system for the text quality metric.

Key Insights and Ideas

  • Integrating Text Quality is Crucial: The most significant insight is that incorporating a text quality metric, derived from a trained MT system, provides substantial benefits over relying solely on coverage or simple text quality metrics. This is particularly true in low-resource settings.
  • Meteor Outperforms Simpler Metrics: The choice of Meteor for text quality evaluation proved beneficial, likely due to its ability to capture deeper linguistic nuances than metrics like BLEU.
  • Greedy Algorithm with Review Works: The iterative greedy algorithm, with its review step for line removal, effectively balances adding good lines and removing detrimental ones to optimize the combined score.
  • Language-Specific Tuning: The optimal parameters (like language ID thresholds and the τ value for coverage) are expected to vary significantly between language pairs, highlighting the need for language-specific adaptation.

Key Results

  • Improved BLEU Scores: The AFRL submission consistently improved BLEU scores over the Zipporah baseline.
  • For 1M en-ne, improvements ranged from 0.24 to 0.48 BLEU.
  • For 1M en-si, improvements ranged from 0.15 to 0.46 BLEU.
  • For 5M en-ne, improvements were around 0.11 BLEU.
  • For 5M en-si, improvements ranged from 0.32 to 0.44 BLEU.
  • "Quality" vs. "Coverage": The tables show that filtering by "quality" alone (equivalent to choosing F=h) often performed worse than the combined metric, especially for larger subcorpus sizes. Similarly, filtering by "coverage" alone was less effective than the combined approach.
  • Subcorpus Size Impact: The improvements were generally largest and most consistent for the 1M-word tests. The authors suggest that larger sets might struggle to find truly helpful data in noisy corpora, leading to convergence towards a text-quality-only score.

Limitations and Future Work

  • Language ID Thresholds: The language identification thresholds were set empirically and might not be optimal for all language pairs.
  • MT System Design: The choice of MT system (phrase-based Moses) and the text quality metric (Meteor) are specific to this submission. Future work could explore using neural MT systems and other metrics like BEER or chrF.
  • Single Text Quality Metric: Building MT systems for both directions of translation (e.g., en-ne and ne-en) and incorporating both text quality scores could potentially reduce dependence on language ID thresholds and yield better subcorpora.
  • Parameter Sensitivity: The optimal choices for parameters like the coverage metric's τ value and n-gram levels are expected to vary, suggesting further research into adaptive parameter selection.

Summary updated

BibTeX
@inproceedings{erdmann2019quality,
  author = {Grant Erdmann and Jeremy Gwinnup},
  title = {{Quality and Coverage: The AFRL Submission to the WMT19 Parallel Corpus Filtering for Low-Resource Conditions Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure