The University of Cambridge's Machine Translation Systems for WMT18

Felix Stahlberg, Adrià de Gispert, Bill Byrne

Abstract

The University of Cambridge submission to the WMT18 news translation task focuses on the combination of diverse models of translation. We compare recurrent, convolutional, and self-attention-based neural models on German-English, English-German, and Chinese-English. Our final system combines all neural models together with a phrase-based SMT system in an MBR-based scheme. We report small but consistent gains on top of strong Transformer ensembles.

Introduction

In 2018, the field of Neural Machine Translation (NMT) was rapidly consolidating around the Transformer architecture. However, the University of Cambridge's submission to the WMT18 news translation task demonstrated that architectural diversity remains a critical component for state-of-the-art performance. The team addressed two primary challenges: effectively integrating disparate models (such as LSTMs, CNNs, and phrase-based SMT) and overcoming hardware constraints to train large-scale Transformer models.

Methodology

The Cambridge system utilized a multi-pronged approach involving diverse architectures and a sophisticated combination framework.

Diverse Architectures

The researchers trained four distinct types of models to capture different linguistic patterns:

  • Recurrent (LSTM): Capturing sequential dependencies.
  • Convolutional (SliceNet): Utilizing local context through convolutions.
  • Self-Attention (Transformer): Standard Transformer and a variant with relative positional embeddings.

Training Efficiency: Delayed SGD

To simulate the massive batch sizes required for optimal Transformer performance (32k–131k tokens) on limited hardware, the team employed gradient accumulation (Delayed SGD). This technique allowed them to achieve the performance of a 16-GPU cluster using only 4 physical GPUs.

Impact of effective batch size (via delayed SGD) on BLEU scores.

Generalized MBR System Combination

The "secret sauce" of the submission was a Minimum Bayes-Risk (MBR) framework. Instead of simple ensemble averaging, they categorized models into "Full Posterior" (standard NMT) and "MBR-based" models (SMT and Right-to-Left NMT). The latter provided n-gram evidence to bias the final translation toward lower-risk hypotheses, effectively integrating models that generate text differently.

Data and Training

The systems were trained on official WMT18 datasets for English-German (En-De), German-English (De-En), and Chinese-English (Zh-En).

  • En-De / De-En: Approximately 42M sentences, including Common Crawl and filtered ParaCrawl.
  • Zh-En: Approximately 46M sentences, including CWMT and UN v1.0 data.
  • Backtranslation: 20M synthetic sentences were generated for each language pair to significantly boost performance.

Results and Analysis

The final system achieved second place in BLEU scores for both English-German and German-English. A key finding was that while the Transformer is the strongest individual architecture, it still benefits from the "opinions" of LSTMs and CNNs. The errors made by different architectures are often uncorrelated, leading to a more robust ensemble.

Ablation study showing the incremental BLEU gains from combining diverse architectures and SMT.

Surprisingly, the inclusion of a Phrase-Based SMT (PBMT) system provided a small but consistent gain, despite its standalone performance being significantly lower than the NMT models. It served as a useful "sanity check" for the neural models. Furthermore, using Right-to-Left (R2L) models as n-gram evidence in the MBR framework proved more effective than traditional n-best list reranking.

Conclusion

The Cambridge WMT18 system underscores the value of architectural diversity in machine translation. By combining various neural architectures with traditional statistical methods via a generalized MBR framework, the team achieved highly competitive results. While the complexity of such an "ensemble of ensembles" may be high for real-time production, it provides a benchmark for the potential gains achievable through model integration. Future work will likely see a diminishing marginal utility for SMT as NMT continues to advance, but the principle of ensembling diverse "viewpoints" remains robust.

Summary updated

Abstract

The University of Cambridge submission to the WMT18 news translation task focuses on the combination of diverse models of translation. We compare recurrent, convolutional, and self-attention-based neural models on German-English, English-German, and Chinese-English. Our final system combines all neural models together with a phrase-based SMT system in an MBR-based scheme. We report small but consistent gains on top of strong Transformer ensembles.
BibTeX
@inproceedings{stahlberg2018university,
  author = {Felix Stahlberg and Adrià de Gispert and Bill Byrne},
  title = {{The University of Cambridge's Machine Translation Systems for WMT18}},
  booktitle = {WMT (shared Task)},
  year = {2018},
  doi = {10.18653/v1/w18-6427},
}
Figure