Abstract
Document-level Neural Machine Translation (NMT) aims to improve translation coherence by incorporating surrounding context. However, the scarcity of document-level parallel data often limits its effectiveness. This paper investigates "imputing" missing context for sentence-level data. We demonstrate that a simple partial copy heuristic—where missing context is filled by copying the current sentence pair—significantly enhances lexical consistency and performance on long-range linguistic phenomena without requiring additional natural document-level data.
Introduction: The Context-Blindness Problem
Standard NMT systems typically translate sentences in isolation, ignoring the broader document context. This "context-blindness" frequently results in errors that are obvious to human readers, such as inconsistent gender markers, mismatched politeness levels (e.g., tu vs. vous in French), and contradictory terminology within a single paragraph.
While document-level NMT models have been developed to address these issues, they face a significant data bottleneck. Most high-quality parallel corpora consist of isolated sentence pairs. Training context-aware models on a mixture of document-level data and context-less sentence pairs often leads the model to ignore context entirely or become confused by missing information.
Methods: Imputing Missing Context
The authors explore strategies to "fill in the blanks" when document-level context is unavailable in training data. They evaluate three primary imputation strategies:
- Random Context: Appending random, unrelated sentence pairs as context. This serves as a baseline but risks training the model to treat context as irrelevant noise.
- Partial Copy Heuristic: A simple yet effective approach where the model creates context by copying the source-target pair currently being translated, occasionally mixed with random sentences. This encourages the model to utilize the context window for lexical cues.
- Context Generation: A more complex method using a language model to generate plausible preceding sentences in the target language, which are then back-translated to create source-side context.
The underlying architecture is a standard Transformer, modified to accept concatenated sequences of multiple sentences as input.
Datasets and Evaluation
The models were trained on the OpenSubtitles2018 English-Russian dataset (~6M sentence pairs), where only 1.5M pairs have natural document-level context. Evaluation goes beyond standard BLEU scores, utilizing four "challenge sets" designed to measure specific linguistic phenomena:
- Lexical Cohesion: Consistency of named entities.
- Deixis: Correct use of politeness markers.
- Verb Phrase Ellipsis: Correct translation of omitted verbs.
- Morphological Inflection: Correct grammatical casing based on context.
Key Insights
- BLEU vs. Coherence: Models with similar BLEU scores showed vastly different performance on challenge sets, indicating that BLEU is insufficient for evaluating document-level coherence.
- The Danger of Randomness: Using purely random context degrades performance on long-distance linguistic tasks by teaching the model to "gate out" the context window.
- The Copying Advantage: The partial copy heuristic was particularly effective for lexical cohesion. By seeing repeated words in the context, the model learns a strong prior for consistency, effectively acting as a "cache" for terminology.
Key Results
The "Partial Copy" model demonstrated significant improvements over the baseline and other imputation methods across all challenge sets.
Specifically, on the En→Ru lexical cohesion task, the baseline scored 45.9%, while the "Partial Copy" model reached 74.9%. Furthermore, combining context imputation with tagged back-translation (using monolingual data to create synthetic documents) allowed the model to reach 90.1% accuracy on lexical cohesion and 89.6% on deixis.
Limitations and Future Work
While the copy heuristic is highly effective, it remains a heuristic that does not provide natural linguistic context. The Context Generation approach is more principled but currently limited by repetitive outputs and compounding errors from back-translation. Future research may focus on more sophisticated context "imagination" techniques and how these data augmentation schemes interact with specialized document-level architectures.
Summary updated