The MLLP-UPV Supervised Machine Translation Systems for WMT19 News Translation Task

Javier Iranzo-Sánchez, Gonçal V. Garcés Díaz-Munío, Jorge Civera, Alfons JuanACL 2019

TL;DR

The MLLP-UPV team presents their systems for the WMT19 News Translation task, focusing on German-English and German-French translation. They achieved strong results by leveraging extensive data filtering, synthetic data generation via backtranslation, and fine-tuning techniques to adapt models to specific domains.

What Problem Does This Paper Address?

Machine translation (MT) systems, particularly Neural Machine Translation (NMT) models, heavily rely on large amounts of parallel training data. However, real-world data, like the ParaCrawl corpus introduced in WMT, can be noisy and introduce errors, potentially degrading translation quality. Furthermore, adapting general-domain MT models to specific news domains can be challenging due to domain mismatch. This paper addresses how to effectively utilize noisy data and adapt models to improve translation performance in the WMT News Translation task.

Methods

The MLLP-UPV team employed several key techniques:

  • Data Preprocessing: Standard Moses pipeline for normalization, tokenization, and true-casing, followed by Byte Pair Encoding (BPE) with 40K operations learned jointly on source and target languages. They also filtered out infrequent subwords and sentences exceeding 100 subwords.
  • Corpus Filtering: Two methods were explored:
  • LM-based filtering: Using language models to score sentence pairs and select those with the lowest perplexity.
  • Dual Conditional Cross-Entropy filtering: A more sophisticated method using language identification, dual conditional cross-entropy, and cross-entropy difference scores. This method proved superior and was adopted.
  • Synthetic Data Generation (Backtranslation): They generated large amounts of synthetic data using backtranslation, a technique that translates monolingual data into the target language using a reverse-direction MT system. This was done for all language pairs, with specific configurations for German-English, English-German, German-French, and French-German. They also experimented with adding noise to the synthetic source sentences.
  • Model Architecture: The Transformer architecture (Vaswani et al., 2017) was used, specifically the Transformer Base and Transformer Big configurations.
  • Training Techniques: They explored the impact of batch size, using multi-GPU training and gradient accumulation to simulate larger effective batch sizes.
  • Finetuning: To address domain mismatch, they finetuned their models on test sets from previous years (for German-English) or dev sets (for German-French) to adapt them to the target domain.

Datasets

  • Training Data:
    • Standard WMT parallel corpora.
  • ParaCrawl corpus (filtered).
  • Synthetic data generated via backtranslation (millions of sentences for each language pair).
  • Monolingual data from News Crawl (various years).
  • Evaluation Data:
    • WMT19 News Translation Shared Task test sets (newstest2017, newstest2018, newstest2019 for German-English; dev1/dev2 and newstest2019 for German-French).
  • BLEU scores were computed using SacreBLEU.

Key Insights and Ideas

  • The Power of Filtering: The Dual Conditional Cross-Entropy filtering method significantly outperformed LM-based filtering, highlighting the importance of robust data cleaning for noisy corpora like ParaCrawl.
  • Backtranslation is Crucial: Generating large amounts of synthetic data via backtranslation was a key factor in improving performance, especially when combined with domain-specific monolingual data.
  • Finetuning for Domain Adaptation: Finetuning on in-domain data (previous test/dev sets) consistently improved translation quality, demonstrating its effectiveness in bridging the gap between general training data and specific task domains.
  • Batch Size Matters for Transformers: The Transformer architecture benefits significantly from larger batch sizes, achieved through multi-GPU training or gradient accumulation, leading to noticeable BLEU score improvements.
  • Noise in Backtranslation: Adding noise to synthetic source sentences showed mixed results, suggesting that the optimal strategy for noise injection might be complex and require further investigation.

Key Results

  • German → English: The best system achieved 48.0 BLEU on newstest2018 and 40.2 BLEU on newstest2019 after finetuning a model trained with noisy backtranslations.
  • English → German: Finetuning a Transformer Big model resulted in an increase of 2.4 BLEU in newstest2018 and 2.3 BLEU in newstest2019 compared to the non-finetuned version.
  • German → French: Finetuning improved performance by 0.2 BLEU in dev2 and 0.1 BLEU in newstest2019.
  • French → German: Finetuning led to an additional increase of 0.5 BLEU in dev2 and 0.6 BLEU in newstest2019.

Limitations and Future Work

  • Noise in Backtranslation: The paper notes that the addition of noise to synthetic data yielded mixed results, and further work is needed to understand its optimal application.
  • Finetuning and Noise Interaction: The authors suggest exploring the interaction between finetuning and adding noise to data.
  • Optimal Data Filtering and Upsampling: Future work could investigate the optimal amount of filtered data from ParaCrawl and the ideal upsampling ratio for mixing bilingual and synthetic data.
  • Time Constraints: Some aspects, like exploring the optimal amount of filtered data and upsampling ratios, were not fully explored due to time constraints during the WMT19 competition.

Summary updated

BibTeX
@inproceedings{iranzosanchez2019mllp,
  author = {Javier Iranzo-Sánchez and Gonçal V. Garcés Díaz-Munío and Jorge Civera and Alfons Juan},
  title = {{The MLLP-UPV Supervised Machine Translation Systems for WMT19 News Translation Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure