NRC Parallel Corpus Filtering System for WMT 2019

Gabriel Bernier-Colborne, Chi-kiu LoACL 2019

TL;DR

This paper presents the National Research Council Canada's (NRC) approach to the WMT 2019 shared task on parallel corpus filtering. They developed four systems, including a deep Transformer network and an ensemble, to identify high-quality sentence pairs for training machine translation (MT) systems, achieving top rankings in the competition.

What Problem Does This Paper Address?

Training effective machine translation (MT) systems relies heavily on the quality of the parallel corpora used. Noisy parallel data, often scraped from the web, can contain misalignments, fluency errors, or sentences in the wrong language, all of which degrade MT performance. The WMT 2019 shared task specifically focused on filtering noisy parallel corpora for low-resource languages (Nepali-English and Sinhala-English), aiming to identify sentence pairs that would lead to the best MT systems when trained on clean data. The challenge lies in developing robust methods to score the quality of these sentence pairs, especially when dealing with limited parallel data.

Methods

The NRC team employed a three-stage pipeline for their systems:

  1. Initial Filtering: This stage removes obvious noise, such as duplicate sentences, sentences with a high proportion of numbers or punctuation, sentences detected as being in the wrong language, and overly long sentences.
  2. Sentence Pair Scoring: This is the core of the systems, where different strategies were used to assign a quality score to each sentence pair.
  3. YiSi-2: This metric evaluates cross-lingual semantic textual similarity using bilingual word embeddings. It's a reference-less approach, meaning it doesn't require human-translated references. They used both supervised and unsupervised versions of YiSi-2.
  4. Deep Transformer Network (XLM): They leveraged a pre-trained cross-lingual language model (XLM) and fine-tuned it on a sentence pair classification task. This model learns to distinguish between actual translations and procedurally generated "bad" translations.
  5. Ensemble: One system combined the scores from YiSi-2 and XLM models.
  6. Re-ranking: A final step was applied to improve vocabulary coverage by penalizing sentence pairs that were too similar to previously selected high-scoring pairs, encouraging diversity.

Datasets

The primary dataset used was the noisy parallel data from the ParaCrawl corpus, specifically for Nepali-English and Sinhala-English language pairs. They also utilized provided small sets of parallel data and larger monolingual corpora for training word embeddings and the XLM model. For evaluation, they used a test set of Wikipedia translations.

Key Insights and Ideas

  • Cross-lingual semantic similarity is crucial: The success of YiSi-2 highlights the importance of measuring how well sentences in different languages convey the same meaning, even if word order or phrasing differs.
  • Pre-trained language models generalize well: The XLM model, pre-trained on vast amounts of multilingual text, demonstrated strong performance when fine-tuned for the sentence pair classification task, showcasing the power of transfer learning.
  • Negative examples are vital for training: Generating "bad" translations (negative examples) was essential for training the XLM model to effectively distinguish between good and poor sentence pairs.
  • Ensembling improves robustness: Combining multiple scoring methods, as done in their ensemble system, led to more consistent and robust performance across different settings.
  • Data size trade-offs: For SMT, larger training samples (5M words) generally yielded better results, suggesting SMT is more tolerant to noise. For NMT, smaller samples (1M words) were often better, indicating NMT is more sensitive to noise in larger datasets.

Key Results

The NRC's systems achieved strong results in the WMT 2019 parallel corpus filtering task, with their best systems ranking 3rd or 4th out of over 20 submissions in most test settings. The ensemble system was particularly robust, providing the most consistent performance. Their SMT systems trained on data selected by their filters achieved competitive BLEU scores, sometimes outperforming baselines like Zipporah, especially with 1M-word samples.

Limitations and Future Work

The authors acknowledge several limitations:

  • Resource Constraints: They had to use a smaller XLM model and had limited time for pre-training and fine-tuning, which may have impacted performance.
  • Suboptimal Model Selection: Evaluating MT quality on only a 1M-word sample of Sinhala before the deadline might have led to suboptimal model choices.
  • Domain Bias: For the Nepali-English pair, the top-scoring sentences often contained biblical texts, suggesting a potential domain bias in their filtering.

Future work could involve:

  • Improved Re-ranking: Developing more sophisticated re-ranking strategies to better optimize vocabulary coverage and MT performance.
  • Self-Training: Adapting the Transformer network to the test data by iteratively selecting likely good examples and updating the model.
  • Further Experimentation: Conducting more extensive testing to assess the usefulness of multi-task fine-tuning and to fully explore the potential of their models.

Summary updated

BibTeX
@inproceedings{berniercolborne2019nrc,
  author = {Gabriel Bernier-Colborne and Chi-kiu Lo},
  title = {{NRC Parallel Corpus Filtering System for WMT 2019}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure