The University of Cambridge's Machine Translation Systems for WMT18

Felix Stahlberg, Adrià de Gispert, Bill Byrne

TL;DR

This paper presents the University of Cambridge's submission to the WMT18 news translation task, focusing on combining diverse neural machine translation (NMT) architectures (recurrent, convolutional, and self-attention) with a traditional phrase-based SMT system. They demonstrate that while self-attention models like the Transformer are superior, combining them with other architectures and even a weaker SMT system through a minimum Bayes-risk (MBR) approach yields significant gains, achieving top results in the competition.

What Problem Does This Paper Address?

The field of machine translation (MT) has seen a rapid rise in Neural Machine Translation (NMT) systems, particularly those based on the Transformer architecture. However, different NMT architectures (like LSTMs, CNNs, and Transformers) and even traditional SMT systems have complementary strengths. The challenge is to effectively combine these diverse models to achieve the best possible translation quality, especially when some individual models might be weaker. This paper tackles the problem of how to best leverage the strengths of multiple, potentially disparate, translation systems to push the state-of-the-art, specifically in the context of the WMT18 news translation task.

Methods

The core of the Cambridge submission lies in its sophisticated system combination strategy. They employ a minimum Bayes-risk (MBR) framework, which is a generalization of previous hybrid NMT-SMT approaches. This method allows them to combine multiple models (up to 10 in their experiments) by treating some as "full posterior" models (contributing their complete translation probability) and others as "MBR-based" models (providing n-gram probabilities).

Key methodological aspects include:

  • Diverse Architectures: They train and evaluate three main neural architectures:
  • Recurrent (LSTM): Standard encoder-decoder LSTMs.
  • Convolutional (SliceNet): Based on Kaiser et al. (2017).
  • Self-Attention (Transformer): Including both the original Transformer and a variant with relative positional embeddings.
  • Right-to-Left Transformer: They also explore a right-to-left Transformer model, not for rescoring, but to extract n-gram posteriors for the MBR combination.
  • Phrase-Based SMT (PBMT): A traditional system is included, which, despite being significantly weaker on its own, proves valuable when combined.
  • MBR Combination: The MBR framework allows for log-linear combination of scores from full-posterior models and n-gram probabilities from MBR-based models. They tune interpolation weights using Powell's method with line search.
  • Delayed SGD Updates: To overcome GPU memory limitations and achieve larger effective batch sizes, they utilize a delayed SGD update technique, which they found crucial for Transformer training and improved BLEU scores.

Datasets

The paper uses standard datasets for the WMT18 news translation task.

  • Training Data: A combination of various parallel corpora including Common Crawl, Europarl, News Commentary, Rapid, ParaCrawl, Synthetic (news-2017), CWMT, UN v1.0, and News Crawl 2017. They apply filtering and over-sampling techniques to these datasets.
  • Development Set: news-test2017 is used for tuning model interpolation weights.
  • Test Sets: Standard WMT test sets for English-German (test14-test18), German-English (test14-test18), and Chinese-English (dev17, test17, test18).

Key Insights and Ideas

  • Self-Attention Dominance, but not Solitude: The Transformer architecture (self-attention) clearly outperforms LSTMs and Convolutional models on its own. However, it still benefits from being combined with other architectures.
  • The Power of MBR for Diversity: The MBR combination scheme is highly effective at integrating diverse models, even those with significantly different performance levels (like the PBMT system). This suggests that complementary strengths can be unlocked even from weaker systems.
  • SMT's Lingering Value: It's surprising that a PBMT system, which scores more than 10 BLEU points below the best neural models, still provides gains when integrated via MBR. This highlights that different systems might capture different linguistic phenomena.
  • Large Batch Sizes are Crucial for Transformers: The paper strongly supports the idea that large batch sizes are essential for training Transformer models effectively, and techniques like delayed SGD are vital for achieving this on limited hardware.
  • Right-to-Left Models Offer N-gram Information: Even if not directly used for rescoring, right-to-left models can contribute valuable n-gram posteriors to an MBR combination.

Key Results

  • The Cambridge submission achieved second place in terms of BLEU score in the WMT18 evaluation campaign for English-German and German-English.
  • Their system outperformed all other systems on a variety of linguistic phenomena for German-English.
  • Combining multiple architectures (Transformer, Relative Transformer, LSTM, SliceNet) and the PBMT system using MBR led to consistent gains over single architectures or simple ensembles.
  • The best system combination (row 10 in Table 7) achieved 31.7 BLEU on English-German news-test17 and 38.7 BLEU on German-English news-test17.
  • Using delayed SGD updates allowed them to scale up the effective number of GPUs and improve BLEU scores significantly (e.g., 29.5 vs. 30.3).

Limitations and Future Work

  • Tuning Complexity: Tuning the interpolation weights for the MBR combination proved challenging, with standard MERT/lattice MERT being ineffective. They resorted to Powell's method, suggesting that more robust tuning strategies might be beneficial.
  • SMT Baseline Weakness: While the PBMT system provided gains, its low individual performance still represents a significant gap. Future work could explore ways to improve the SMT baseline or integrate it more effectively.
  • Computational Cost: Training and combining multiple large neural models is computationally intensive. While they addressed GPU memory limitations with delayed SGD, further optimizations for efficiency are always desirable.
  • Future Directions: The paper implicitly suggests that exploring more sophisticated combination strategies, further investigating the complementary strengths of different architectures, and leveraging larger datasets or backtranslation remain promising avenues for future research.

Summary updated

BibTeX
@inproceedings{stahlberg2018university,
  author = {Felix Stahlberg and Adrià de Gispert and Bill Byrne},
  title = {{The University of Cambridge's Machine Translation Systems for WMT18}},
  booktitle = {WMT (shared Task)},
  year = {2018},
  doi = {10.18653/v1/w18-6427},
}
Figure