When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion

Elena Voita, Rico Sennrich, Ivan Titov

Abstract

Though machine translation errors caused by the lack of context beyond one sentence have long been acknowledged, the development of context-aware NMT systems is hampered by several problems. Firstly, standard metrics are not sensitive to improvements in consistency in document-level translations. Secondly, previous work on context-aware NMT assumed that the sentence-aligned parallel data consisted of complete documents while in most practical scenarios such document-level data constitutes only a fraction of the available parallel data. To address the first issue, we perform a human study on an English-Russian subtitles dataset and identify deixis, ellipsis and lexical cohesion as three main sources of inconsistency. We then create test sets targeting these phenomena. To address the second shortcoming, we consider a set-up in which a much larger amount of sentence-level data is available compared to that aligned at the document level. We introduce a model that is suitable for this scenario and demonstrate major gains over a context-agnostic baseline on our new benchmarks without sacrificing performance as measured with BLEU.

Introduction

Most Neural Machine Translation (NMT) systems are "context-agnostic," translating one sentence at a time. While effective for isolated snippets, this approach frequently fails at the document level. A system might inconsistently translate the formal/informal "you" (deixis) or fail to correctly inflect a noun because it lacks context for an elided verb from a previous sentence. This paper addresses two major hurdles: the "metric blindness" of standard tools like BLEU, which are insensitive to document-level consistency, and the scarcity of document-aligned parallel data compared to sentence-aligned data.

The Context-Aware Decoder (CADec)

The authors propose a two-pass framework inspired by deliberation networks. The first pass utilizes a standard Transformer-base model trained on a massive 6-million sentence dataset to ensure high general translation quality. The second pass, the Context-Aware Decoder (CADec), then refines this output.

The CADec architecture uses a two-pass approach to refine translations using context.

CADec employs a specialized attention mechanism to incorporate the source and target context of the three preceding sentences. To prevent the model from simply copying the first pass, a denoising objective is used during training; the model is fed either the base model's output or a corrupted reference translation, forcing it to learn how to resolve contextual ambiguities.

Targeted Evaluation

The study focuses on English-Russian translation using the OpenSubtitles2018 corpus. To move beyond standard metrics, the authors introduced new contrastive test sets (6,000 examples) designed to evaluate three specific phenomena:

  • Deixis: Specifically the T-V distinction and gender consistency.
  • Ellipsis: Correctly predicting morphological forms based on previous sentences.
  • Lexical Cohesion: Ensuring named entities are translated consistently across a dialogue.

In these tests, the model is tasked with ranking a contextually "correct" translation against an "incorrect" one that is plausible only in isolation.

Results and Insights

CADec achieved major gains in consistency without sacrificing general translation quality. While the BLEU score remained statistically indistinguishable from the baseline (32.38 vs 32.40), accuracy on the deixis test set jumped from 50.0% to 81.6%, and Verb Phrase ellipsis accuracy rose from 28.4% to 80.0%.

Comparison of BLEU score and accuracy on targeted test sets during training. While BLEU plateaus, accuracy on specific linguistic phenomena continues to improve.

A key insight from the training dynamics is that BLEU scores often plateau early, while accuracy on context-specific test sets continues to improve. This highlights the necessity of targeted linguistic benchmarks to measure progress in document-level NMT.

Conclusion

By separating the translation task from contextual refinement, CADec successfully leverages large-scale sentence-level data alongside smaller document-level subsets. While currently limited to a three-sentence context window and specific language pairs, the model demonstrates a robust path forward for fixing the inconsistencies that context-agnostic models introduce.

Summary updated

Abstract

Though machine translation errors caused by the lack of context beyond one sentence have long been acknowledged, the development of context-aware NMT systems is hampered by several problems. Firstly, standard metrics are not sensitive to improvements in consistency in document-level translations. Secondly, previous work on context-aware NMT assumed that the sentence-aligned parallel data consisted of complete documents while in most practical scenarios such document-level data constitutes only a fraction of the available parallel data. To address the first issue, we perform a human study on an English-Russian subtitles dataset and identify deixis, ellipsis and lexical cohesion as three main sources of inconsistency. We then create test sets targeting these phenomena. To address the second shortcoming, we consider a set-up in which a much larger amount of sentence-level data is available compared to that aligned at the document level. We introduce a model that is suitable for this scenario and demonstrate major gains over a context-agnostic baseline on our new benchmarks without sacrificing performance as measured with BLEU.
BibTeX
@inproceedings{voita2019when,
  author = {Elena Voita and Rico Sennrich and Ivan Titov},
  title = {{When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=L_lSN65HLaZ},
}
Figure