Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation

Marcin Junczys-DowmuntACL 2019

TL;DR

This paper tackles the challenge of improving neural machine translation (NMT) by moving beyond sentence-by-sentence translation to consider entire documents. The authors demonstrate that by training models on longer sequences and incorporating document-level context, they achieve significant improvements in translation quality, particularly for the English-German language pair.

What Problem Does This Paper Address?

Traditional neural machine translation systems process text one sentence at a time. While this approach has achieved remarkable success, it often fails to capture the broader context that spans across sentences within a document. This can lead to inconsistencies in terminology, pronoun resolution errors, and a general lack of coherence in the translated output. The WMT (Workshop on Machine Translation) shared task provides a benchmark for evaluating these systems, and this paper aims to push the boundaries of NMT by developing models that can leverage document-level information for better translation quality. The core problem is how to effectively incorporate this longer-range context into NMT models without sacrificing performance or introducing new issues.

Methods

The Microsoft Translator team employed a multi-pronged approach to develop their document-level NMT systems:

  • Deep Transformer Models: They utilized the Transformer architecture, a standard in NMT, but explored deeper variants (12-layer models instead of the typical 6-layer) to better handle the increased complexity of document-level context.
  • Longer Sequence Training: A key innovation was training models on sequences of up to 1000 subword segments, effectively allowing them to process multiple sentences or even entire short documents at once.
  • Data Augmentation and Filtering:
    • Data Filtering: They used techniques to filter noisy parallel data, improving the quality of training material.
  • Noisy Back-Translation: They generated large amounts of synthetic parallel data by back-translating monolingual text, a common and effective technique for boosting NMT performance.
  • Document Boundary Restoration: For some datasets, they restored document boundaries that were previously lost, enabling the training of document-aware models.
  • Synthetic Documents: They created "fake" documents by randomly concatenating parallel sentences to simulate document-level context where natural boundaries were unavailable.
  • Fine-Tuning: Models were fine-tuned on specific data subsets to optimize performance on different aspects of the translation task, particularly to address quality drops on the original English side.
  • Multi-Task Learning with BERT: They incorporated a BERT-style masked language modeling (MLM) objective for the encoder, trained on monolingual English documents. This aimed to improve the encoder's understanding of source language context.
  • Ensembling: Various combinations of trained models (sentence-level and document-level) were ensembled to further boost performance. They explored both homogeneous (similar models) and heterogeneous (diverse models) ensembles.
  • Second-Pass Decoding: They experimented with a two-stage decoding process, where a sentence-level translation was first generated and then refined using document-level context.

Datasets

The paper utilized several datasets for training and evaluation:

  • WMT19 News Translation Shared Task Data: This served as the primary benchmark for evaluation, specifically the English-German pair.
  • Europarl, Rapid, News-Commentary: Portions of these datasets had their document boundaries restored for training.
  • Monolingual News Crawl Data: This large corpus was used for back-translation and for training the BERT encoder.
  • Synthetic Parallel Data: Generated through back-translation and random document boundary insertion.
  • Test Sets: Original test sets from WMT16, WMT17, and WMT18 were used for evaluation, alongside a newly created development set derived from these.

Key Insights and Ideas

  • Document Context Matters: The most significant insight is that explicitly modeling document-level context leads to substantial improvements in translation quality, outperforming comparable sentence-level systems.
  • Back-Translation's Role: Back-translation proved beneficial, particularly for "translationese" input (text that sounds like a translation), but its effectiveness could be further enhanced by fine-tuning.
  • Deeper Models Benefit More: Deeper Transformer models (12-layer) showed greater gains from the additional context provided by longer sequences.
  • Fine-tuning Recovers Quality: Fine-tuning on the original English side was crucial for restoring and even improving translation quality on that specific split, without significantly sacrificing gains on the German side.
  • BERT Encoder Boosts Contextual Understanding: Multi-task training with a BERT encoder on monolingual data showed promise in improving the source language understanding for translation.
  • Ensembling is Key: Combining diverse models, especially those trained with document-level context, yielded the best results, though careful weighting was needed to balance performance across different evaluation splits.

Key Results

  • Human Evaluation Preference: Human evaluators strongly preferred the document-level systems over comparable sentence-level systems.
  • Source-Based Direct Assessment: Document-level systems also scored higher than human references in source-based direct assessment.
  • BLEU Score Improvements: While specific BLEU scores are detailed in the tables, the paper consistently shows improvements for document-level systems over sentence-level baselines, particularly on the "separated by origin" splits of the development set. For example, the best document-level ensemble achieved a BLEU score of 45.2 on the "en" split of the dev set, compared to 43.8 for the best sentence-level ensemble.
  • Long Sequences Don't Degrade Performance: Training on very long sequences (up to 1000 tokens) did not degrade translation performance compared to sentence-level systems.

Limitations and Future Work

  • Hyper-parameter Tuning for Deeper Models: The authors noted that they did not see further improvements from even deeper models (18 and 24 blocks), suggesting potential issues with hyper-parameter tuning for these very deep architectures.
  • Weighting Ensembles: While they found success in down-weighting type (a) models (trained only on parallel data) in ensembles, they acknowledge that an ensemble of only type (c) models (trained with back-translation and fine-tuned) might have been a better choice if more such models were trained.
  • Sentence Embeddings: The paper mentions that sentence embeddings were not used but remain an area for future exploration.
  • Ablation Studies: The authors emphasize the need for more rigorous ablation testing to confirm the precise contributions of different components.
  • Final Ranking Uncertainty: They express some doubt about the preliminary ranking in Table 7, particularly the discrepancy between raw scores and normalized z-scores, suggesting that raw scores are a more reliable indicator for deployment decisions.

Summary updated

BibTeX
@inproceedings{junczysdowmunt2019microsoft,
  author = {Marcin Junczys-Dowmunt},
  title = {{Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure