When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion

Elena Voita, Rico Sennrich, Ivan Titov

Abstract

Standard neural machine translation (NMT) systems often translate sentences in isolation, leading to document-level inconsistencies. This research identifies three primary linguistic sources of these errors—deixis, ellipsis, and lexical cohesion—and introduces CADec (Context-Aware Decoder), a novel two-pass model. CADec improves consistency on targeted benchmarks without sacrificing general translation quality, even when document-level training data is limited.

The Problem: Context-Agnostic Translation

Most modern NMT systems are context-agnostic, meaning they process one sentence at a time. While this approach produces grammatically correct individual sentences, it often fails at the document level. For instance, a system might inconsistently translate the pronoun "you" between formal and informal forms in Russian (the T-V distinction) or use different translations for the same entity name across a single scene.

Two major hurdles have historically hindered the development of context-aware NMT:

  1. The Evaluation Gap: Standard metrics like BLEU are insensitive to document-level consistency. A model can significantly improve logic and cohesion while maintaining the same BLEU score.
  2. Data Scarcity: Context-aware models typically require massive document-level parallel corpora, which are far rarer than individual parallel sentence pairs.

Methodology: The CADec Architecture

To address these issues, the authors propose CADec, a two-pass framework designed to refine translations using surrounding context.

The CADec architecture: a two-pass framework using a base model for initial translation and a context-aware stage for refinement.
  • Pass 1: A standard, context-agnostic Transformer (the "base model") generates an initial translation.
  • Pass 2: The CADec model takes this initial output and refines it by attending to the context of the three preceding source and target sentences.

To train CADec effectively on limited document-level data, the authors employed a "denoising" strategy. During training, the model was fed either the base model's translation or a corrupted version of the reference translation. This taught the model to rely on the first pass for general meaning while looking to the context to fix specific consistency errors.

Training and the BLEU Paradox

A key insight from this research is that standard stopping criteria based on BLEU can be misleading for context-aware models. As shown in the training curves below, while the general BLEU score plateaus or even slightly declines after a certain point, the model's accuracy on specific linguistic phenomena (like lexical cohesion) continues to improve.

Comparison of BLEU score and targeted accuracy during training. While BLEU plateaus early, the model's ability to handle context-dependent phenomena continues to rise.

Key Results

The study utilized the English-Russian OpenSubtitles2018 corpus and introduced a new targeted test suite of 6,000 examples. CADec demonstrated substantial improvements across all targeted categories:

  • General Quality: CADec maintained a BLEU score of 32.38, nearly identical to the context-agnostic baseline (32.40).
  • Deixis: Accuracy in handling pronouns and politeness levels rose from 50.0% to 81.6%.
  • Lexical Cohesion: Accuracy in maintaining consistent entity naming improved from 45.9% to 58.1%.
  • Ellipsis: In verb phrase ellipsis tasks, accuracy jumped from 28.4% to 80.0%.

While simpler "concatenation" baselines (which feed context directly into the input) showed some improvements, they often suffered a significant drop in overall BLEU score (~0.8 points). CADec’s two-pass approach avoids this trade-off.

Conclusion

CADec proves that context-aware translation can be achieved without massive document-level datasets or sacrificing general translation quality. By focusing on the "Big Three" of inconsistency—deixis, ellipsis, and lexical cohesion—this model moves NMT closer to human-level document understanding. Future work may involve expanding these techniques to other language pairs and exploring dynamic context windows for longer-range dependencies.

Summary updated

Abstract

Though machine translation errors caused by the lack of context beyond one sentence have long been acknowledged, the development of context-aware NMT systems is hampered by several problems. Firstly, standard metrics are not sensitive to improvements in consistency in document-level translations. Secondly, previous work on context-aware NMT assumed that the sentence-aligned parallel data consisted of complete documents while in most practical scenarios such document-level data constitutes only a fraction of the available parallel data. To address the first issue, we perform a human study on an English-Russian subtitles dataset and identify deixis, ellipsis and lexical cohesion as three main sources of inconsistency. We then create test sets targeting these phenomena. To address the second shortcoming, we consider a set-up in which a much larger amount of sentence-level data is available compared to that aligned at the document level. We introduce a model that is suitable for this scenario and demonstrate major gains over a context-agnostic baseline on our new benchmarks without sacrificing performance as measured with BLEU.
BibTeX
@inproceedings{voita2019when,
  author = {Elena Voita and Rico Sennrich and Ivan Titov},
  title = {{When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=L_lSN65HLaZ},
}
Figure