TL;DR
Researchers from Charles University improved English-Czech machine translation by moving beyond isolated sentences to document-level context. By training Transformer models on multi-sentence segments, they achieved a modest but statistically significant +0.6 BLEU improvement and demonstrated how broader context can resolve tricky lexical ambiguities that single-sentence models miss.
What Problem Does This Paper Address?
Standard Neural Machine Translation (NMT) typically treats a document as a bag of independent sentences. While this works surprisingly well, it ignores inter-sentential phenomena like pronoun resolution, consistent tense usage, and lexical coherence. For example, a polysemous English word like "goal" might be translated as a sports score in one sentence and a life ambition in the next, even if the document is clearly about football.
Human evaluators often find that while NMT systems approach "human parity" on isolated sentences, they fall significantly short when evaluated on whole documents. The authors aimed to bridge this gap for the WMT19 English-Czech news translation task by providing the model with surrounding context during both training and inference.
Methods
The authors built their systems using two popular NMT frameworks: Tensor2Tensor (T2T) and Marian. Their core innovation was Training Data Context Augmentation. Instead of feeding the model single sentence pairs, they extracted sequences of consecutive sentences up to a limit of 1000 characters (averaging about 15 sentences).
To handle these longer sequences during translation (decoding), they employed two different strategies:
- Sliding Window (Marian): They translated 3-sentence blocks and primarily used the translation of the middle sentence, which benefited from both preceding and succeeding context.
- Overlapping Splits (T2T): They defined "pre-context" and "post-context" sentences that were fed into the model to provide background but were discarded in the final output, keeping only the "main content" translations.
They also focused on reimplementing their successful WMT18 system in the Marian framework to compare performance across different engineering stacks.
Datasets
The models were trained on a massive collection of parallel and synthetic data allowed for the WMT19 shared task, totaling approximately 184 million sentence pairs.
Key sources included:
- CzEng 1.7: The primary parallel corpus.
- Europarl, News Commentary, CommonCrawl, and WikiTitles.
- Backtranslated data: Large amounts of synthetic data created by translating Czech news mono-data back into English.
Key Insights and Ideas
- Context Resolves Ambiguity: The most significant "aha" moment was seeing the document-level model correctly choose between non-synonymous Czech lemmata. For instance, the word "goal" was correctly translated as gól (sports) rather than cíl (aim) when the surrounding sentences mentioned sports context.
- Framework Parity is Hard: Despite using identical data and similar hyperparameters, the T2T implementation outperformed the Marian implementation by about 1.7 BLEU points. The authors hypothesize this is due to subtle differences in optimizers (Adafactor vs. Adam) and batching strategies.
- The "Weak Evidence" Challenge: While document-level models feel intuitively better, proving it quantitatively is difficult. The authors found that cases where inter-sentential context is absolutely necessary for correct translation are relatively rare in standard news datasets, making it hard to draw sweeping conclusions from BLEU scores alone.
Key Results
The DocTransformer T2T system emerged as the top performer among their submissions.
- BLEU Gains: The document-level T2T system achieved 31.03 BLEU (uncased), a +0.6 improvement over applying the same model to isolated sentences.
- Qualitative Success: Manual analysis confirmed that the document-level models corrected specific lexical errors. As shown in the appendix, the document-level models were better at identifying that "running" referred to a political candidacy rather than physical movement, and "settlement" referred to a legal compensation rather than a village.
Limitations and Future Work
The authors noted several limitations:
- Efficiency: Decoding with large contexts is computationally expensive and time-consuming.
- Sampling Bias: Their context augmentation method implicitly upsamples sentences from longer documents, an effect they leave for future analysis.
- Evaluation Metrics: Standard metrics like BLEU are not sensitive enough to capture the subtle improvements in document coherence.
Future work involves exploring more sophisticated sampling methods and finding better ways to evaluate inter-sentential consistency beyond manual spot-checks.
Summary updated