The JHU Machine Translation Systems for WMT 2018

Philipp Koehn, Kevin Duh, Brian ThompsonWMT (shared Task) 2018 · OpenReview

TL;DR

Johns Hopkins University developed neural machine translation (NMT) systems for the WMT 2018 shared task, focusing on German-English, English-German, and Russian-English. Their key innovations were iterative back-translation and fine-tuning on previous years' test sets, leading to improved performance, especially for the challenging Russian-English pair.

What Problem Does This Paper Address?

The paper addresses the ongoing challenge of building high-quality neural machine translation systems, particularly for news translation tasks. While NMT models have become state-of-the-art, achieving top performance requires significant computational resources and careful tuning. The WMT shared task provides a benchmark for comparing different approaches. The authors specifically aim to improve translation quality by exploring techniques like iterative back-translation and leveraging data from previous years' tasks, which can be particularly useful when new parallel data is scarce or expensive to acquire. They also investigate the impact of different model architectures and training strategies on performance.

Methods

The core of the JHU systems relies on the Marian and Sockeye NMT toolkits, which implement recurrent neural network (RNN) based architectures, specifically LSTMs. Their novel contributions include:

  • Iterative Back-Translation: This technique involves using a trained NMT system to translate monolingual data into the target language, creating synthetic parallel data. This synthetic data is then used to retrain or fine-tune the NMT system, iteratively improving its performance. They explored a two-round iterative process.
  • Fine-tuning on Prior Test Sets: The authors fine-tuned their models on test sets from previous WMT shared tasks. This is a form of domain adaptation, leveraging data that is similar in domain to the target test set.
  • Ensembling: They employed various ensembling techniques, including checkpoint ensembling (averaging parameters from multiple checkpoints of a single training run) and ensembling of independent training runs.
  • Hyperparameter Search: Extensive hyperparameter tuning was conducted, exploring variations in LSTM layers, embedding dimensions, hidden units, and dropout rates.
  • Right-to-Left Reranking: For German-English, they also experimented with reranking using models trained on reversed sentence order.

Datasets

  • Parallel Data: Standard parallel corpora for German-English, English-German, and Russian-English were used for initial training.
  • Monolingual Data: Monolingual news text was utilized for back-translation.
  • Paracrawl Corpus: For German-English, they experimented with adding the unfiltered Paracrawl corpus to the training data.
  • Prior Test Sets: Test sets from previous WMT shared tasks (e.g., newstest2012-2016) were used for fine-tuning and domain adaptation.
  • Evaluation Sets: newstest2017 was used as the primary test set for reporting results. newstest2016 was used as the development set for early stopping.

Key Insights and Ideas

  • Iterative Back-Translation Pays Off: The paper strongly suggests that iterative back-translation significantly boosts translation quality, especially when combined with deep model architectures and ensembling.
  • Leveraging Past Data is Crucial: Fine-tuning on data from previous years' test sets proved to be an effective strategy for domain adaptation and improving performance on the current test set.
  • Ensembling is Key for Top Performance: Combining multiple models (through checkpoint ensembling and ensembling of independent runs) consistently yielded substantial gains over single models.
  • Perplexity as a Surrogate for BLEU: The authors found that perplexity (both training and development) correlated well with BLEU scores during hyperparameter search, suggesting it can be a useful proxy for evaluating models when BLEU computation is expensive.
  • Training Time Matters: Longer training times generally led to better performance, with models showing continuous improvement even late in the training process.
  • Subword Units Impact Hyperparameter Search: Experiments with different numbers of BPE merge operations indicated that hyperparameters optimized for one BPE configuration might not transfer directly to another.

Key Results

  • German-English: The JHU systems outperformed the previous year's best results, achieving 37.0 BLEU on German-English and 29.1 BLEU on English-German. However, they fell short of the top-performing Transformer-based systems in the current year.
  • Russian-English: The paper highlights significant improvements for the Russian-English pair, with iterative back-translation and fine-tuning leading to substantial gains.
  • Paracrawl Impact: Adding the unfiltered Paracrawl corpus to the German-English system resulted in a score of 36.3 BLEU, an improvement over systems without it.
  • Continued Training Gains: Fine-tuning models for a few epochs on adaptation data (newstest2012-2016) improved BLEU scores by 0.6 to 1.9 points.
  • Ensemble Performance: A 6-model ensemble achieved 33.63 BLEU for Russian-English, a notable improvement.

Limitations and Future Work

  • Transformer vs. RNN: The authors acknowledge that their RNN-based models were outperformed by Transformer models in the WMT 2018 competition.
  • Computational Constraints: Long training times (e.g., a month on a single GPU) limited their ability to fully exploit large corpora like Paracrawl and explore more extensive hyperparameter spaces.
  • Subword Unit Sensitivity: The impact of different BPE configurations on hyperparameter search warrants further investigation.
  • Future Work: The paper suggests that further gains might be achieved by adjusting learning rates, convergence criteria, and exploring more robust hyperparameter search strategies. They also hint at the need for enhancements beyond standard sequence-to-sequence models if significant further improvements are to be made, potentially by incorporating techniques like reranking or SMT/NMT hybrids.

Summary updated

Abstract

We report on the efforts of the Johns Hopkins University to develop neural machine translation systems for the shared task for news translation organized around the Conference for Machine Translation (WMT) 2018. We developed systems for German–English, English– German, and Russian–English. Our novel contributions are iterative back-translation and fine-tuning on test sets from prior years.
BibTeX
@inproceedings{koehn2018jhu,
  author = {Philipp Koehn and Kevin Duh and Brian Thompson},
  title = {{The JHU Machine Translation Systems for WMT 2018}},
  booktitle = {WMT (shared Task)},
  year = {2018},
  url = {https://openreview.net/forum?id=dg1vdwRT_KV},
}
Figure