TL;DR
This paper tackles the challenge of improving neural machine translation (NMT) by moving beyond sentence-by-sentence translation to consider entire documents. The authors demonstrate that by training models on longer sequences and incorporating document-level context, they achieve significant improvements in translation quality, particularly for the English-German language pair.
What Problem Does This Paper Address?
Traditional neural machine translation systems process text one sentence at a time. While this approach has achieved remarkable success, it often fails to capture the broader context that spans across sentences within a document. This can lead to inconsistencies in terminology, pronoun resolution errors, and a general lack of coherence in the translated output. The WMT (Workshop on Machine Translation) shared task provides a benchmark for evaluating these systems, and this paper aims to push the boundaries of NMT by developing models that can leverage document-level information for better translation quality. The core problem is how to effectively incorporate this longer-range context into NMT models without sacrificing performance or introducing new issues.
Methods
The Microsoft Translator team employed a multi-pronged approach to develop their document-level NMT systems:
- Deep Transformer Models: They utilized the Transformer architecture, a standard in NMT, but explored deeper variants (12-layer models instead of the typical 6-layer) to better handle the increased complexity of document-level context.
- Longer Sequence Training: A key innovation was training models on sequences of up to 1000 subword segments, effectively allowing them to process multiple sentences or even entire short documents at once.
- Data Augmentation and Filtering:
- Data Filtering: They used techniques to filter noisy parallel data, improving the quality of training material.
- Noisy Back-Translation: They generated large amounts of synthetic parallel data by back-translating monolingual text, a common and effective technique for boosting NMT performance.
- Document Boundary Restoration: For some datasets, they restored document boundaries that were previously lost, enabling the training of document-aware models.
- Synthetic Documents: They created "fake" documents by randomly concatenating parallel sentences to simulate document-level context where natural boundaries were unavailable.
- Fine-Tuning: Models were fine-tuned on specific data subsets to optimize performance on different aspects of the translation task, particularly to address quality drops on the original English side.
- Multi-Task Learning with BERT: They incorporated a BERT-style masked language modeling (MLM) objective for the encoder, trained on monolingual English documents. This aimed to improve the encoder's understanding of source language context.
- Ensembling: Various combinations of trained models (sentence-level and document-level) were ensembled to further boost performance. They explored both homogeneous (similar models) and heterogeneous (diverse models) ensembles.
- Second-Pass Decoding: They experimented with a two-stage decoding process, where a sentence-level translation was first generated and then refined using document-level context.
Datasets
The paper utilized several datasets for training and evaluation:
- WMT19 News Translation Shared Task Data: This served as the primary benchmark for evaluation, specifically the English-German pair.
- Europarl, Rapid, News-Commentary: Portions of these datasets had their document boundaries restored for training.
- Monolingual News Crawl Data: This large corpus was used for back-translation and for training the BERT encoder.
- Synthetic Parallel Data: Generated through back-translation and random document boundary insertion.
- Test Sets: Original test sets from WMT16, WMT17, and WMT18 were used for evaluation, alongside a newly created development set derived from these.
Key Insights and Ideas
- Document Context Matters: The most significant insight is that explicitly modeling document-level context leads to substantial improvements in translation quality, outperforming comparable sentence-level systems.
- Back-Translation's Role: Back-translation proved beneficial, particularly for "translationese" input (text that sounds like a translation), but its effectiveness could be further enhanced by fine-tuning.
- Deeper Models Benefit More: Deeper Transformer models (12-layer) showed greater gains from the additional context provided by longer sequences.
- Fine-tuning Recovers Quality: Fine-tuning on the original English side was crucial for restoring and even improving translation quality on that specific split, without significantly sacrificing gains on the German side.
- BERT Encoder Boosts Contextual Understanding: Multi-task training with a BERT encoder on monolingual data showed promise in improving the source language understanding for translation.
- Ensembling is Key: Combining diverse models, especially those trained with document-level context, yielded the best results, though careful weighting was needed to balance performance across different evaluation splits.
Key Results
- Human Evaluation Preference: Human evaluators strongly preferred the document-level systems over comparable sentence-level systems.
- Source-Based Direct Assessment: Document-level systems also scored higher than human references in source-based direct assessment.
- BLEU Score Improvements: While specific BLEU scores are detailed in the tables, the paper consistently shows improvements for document-level systems over sentence-level baselines, particularly on the "separated by origin" splits of the development set. For example, the best document-level ensemble achieved a BLEU score of 45.2 on the "en" split of the dev set, compared to 43.8 for the best sentence-level ensemble.
- Long Sequences Don't Degrade Performance: Training on very long sequences (up to 1000 tokens) did not degrade translation performance compared to sentence-level systems.
Limitations and Future Work
- Hyper-parameter Tuning for Deeper Models: The authors noted that they did not see further improvements from even deeper models (18 and 24 blocks), suggesting potential issues with hyper-parameter tuning for these very deep architectures.
- Weighting Ensembles: While they found success in down-weighting type (a) models (trained only on parallel data) in ensembles, they acknowledge that an ensemble of only type (c) models (trained with back-translation and fine-tuned) might have been a better choice if more such models were trained.
- Sentence Embeddings: The paper mentions that sentence embeddings were not used but remain an area for future exploration.
- Ablation Studies: The authors emphasize the need for more rigorous ablation testing to confirm the precise contributions of different components.
- Final Ranking Uncertainty: They express some doubt about the preliminary ranking in Table 7, particularly the discrepancy between raw scores and normalized z-scores, suggesting that raw scores are a more reliable indicator for deployment decisions.
Summary updated