TL;DR
This survey provides a comprehensive overview of document-level Neural Machine Translation (NMT), a field that aims to improve translation quality by considering context beyond individual sentences. It categorizes existing research based on how context is modeled, learned, and evaluated, highlighting the growing importance of this area and suggesting future research directions.
What Problem Does This Paper Address?
Traditional Neural Machine Translation (NMT) models translate sentences in isolation, ignoring the rich interdependencies and discourse structures that exist within a larger document. This sentence-level approach can lead to inconsistencies, ambiguous translations, and a loss of intended meaning, especially when dealing with phenomena like pronoun resolution or lexical cohesion that span across sentences. The paper addresses the need for a systematic understanding of how researchers are tackling this challenge by incorporating document-level context into NMT systems.
Methods
The paper organizes the surveyed literature into several key areas:
- Modeling Document-Level Context: This involves how systems incorporate information from surrounding sentences. Approaches are categorized by:
- Local vs. Global Context: Whether they use a few neighboring sentences (local) or the entire document (global).
- Source-only vs. Source and Target Context: Whether context is drawn solely from the source document or from both source and target sides.
- Architectural Novelties: This includes modifications to encoder-decoder architectures, such as adding extra encoders for context, using attention mechanisms to focus on relevant parts of the context, or employing hierarchical structures. The Transformer architecture is a common backbone for these advancements.
- Learning with Document-Level Context: This section discusses how models are trained to leverage document context. It covers:
- Modified Training Objectives: Adjustments to standard NMT training objectives to encourage context utilization, including regularization terms and document-level reward functions.
- Contextualized Word Embeddings: Utilizing pre-trained language models like BERT to provide rich contextual information.
- Decoding with Document-Level Context: This focuses on how the best translation is generated from a document. It includes techniques like:
- Shallow Fusion: Combining a sentence-level NMT model with a document-level language model.
- Two-Pass Decoding: Refining initial sentence-level translations by considering document context.
- Self-Training: Iteratively improving models by using their own outputs.
Datasets
The paper discusses various datasets used in document-level NMT research, including:
- Standard NMT Datasets: Europarl, News-Commentary, and WMT datasets are frequently used, often with document boundaries identified or created.
- Specialized Datasets:
- OpenSubtitles: A large corpus of movie and TV subtitles, often used for its conversational nature and document structure.
- RotoWire: A dataset for sports articles.
- TED Talks Corpus: Used for its multi-domain nature.
- New Benchmarks/Test Suites: The paper highlights the introduction of new test suites and human evaluation setups in shared tasks like WMT19 and WNGT2019, designed to specifically assess document-level phenomena like anaphora resolution, coherence, and cohesion.
Key Insights and Ideas
- Sentence-level NMT is insufficient: The core insight is that ignoring document context leads to translation errors that are invisible at the sentence level but crucial for overall text quality.
- Context is key for discourse phenomena: Phenomena like pronoun resolution (anaphora), lexical cohesion, and coherence are heavily reliant on inter-sentential context.
- Transformer architecture is a strong foundation: Many document-level NMT approaches build upon the Transformer architecture, leveraging its attention mechanisms to incorporate context.
- Local context is more common, but global context is more general: While most research focuses on incorporating a few neighboring sentences, global context approaches are more flexible and can be simplified to local context.
- Evaluation is a major challenge: Standard metrics like BLEU are insufficient for evaluating document-level translation quality, necessitating the development of discourse-targeted test sets and more sophisticated evaluation schemes.
- Pre-trained models offer a significant boost: Leveraging pre-trained contextualized language models (like BERT) has shown considerable promise for improving document-level NMT.
Key Results
- Document-level NMT models have demonstrated significant improvements in BLEU scores and other evaluation metrics compared to sentence-level baselines, particularly for tasks involving discourse phenomena.
- Specific approaches have shown success in improving anaphora resolution, lexical cohesion, and overall coherence.
- The use of context-aware decoders and modified training objectives has led to more consistent and fluent translations.
- Shared tasks like WMT19 and WNGT2019 have spurred innovation, with systems achieving competitive results by incorporating document-level context.
Limitations and Future Work
- Dataset scarcity: A major bottleneck is the lack of large, document-parallel bilingual datasets, especially for specific discourse phenomena and morphologically rich languages. The creation of such resources is a key future direction.
- Evaluation challenges: There is no consensus on document-level MT evaluation metrics. While automatic metrics are improving, they still struggle to capture discourse nuances. More in-depth discourse-targeted analysis and a balance between automatic and manual evaluation are needed.
- Exploiting global context: The true utility of global context remains an open question, requiring more targeted analysis to understand its benefits beyond local context.
- Automating discourse annotation: Developing automated methods for annotating discourse phenomena could significantly advance both the development and evaluation of document-level NMT.
- Morphologically rich languages and multi-domain translation: Further research is needed to test document-level NMT approaches on a wider range of languages and domains.
Summary updated