Fill in the Blanks: Imputing Missing Sentences for Larger-Context Neural Machine Translation

Sébastien Jean, Ankur Bapna, Orhan FiratarXiv 2019 · arXiv

Abstract

Document-level Neural Machine Translation (NMT) aims to improve translation coherence by incorporating surrounding context. However, the scarcity of document-level parallel data often limits its effectiveness. This paper investigates "imputing" missing context for sentence-level data. We demonstrate that a simple partial copy heuristic—where missing context is filled by copying the current sentence pair—significantly enhances lexical consistency and performance on long-range linguistic phenomena without requiring additional natural document-level data.

Introduction: The Context-Blindness Problem

Standard NMT systems typically translate sentences in isolation, ignoring the broader document context. This "context-blindness" frequently results in errors that are obvious to human readers, such as inconsistent gender markers, mismatched politeness levels (e.g., tu vs. vous in French), and contradictory terminology within a single paragraph.

While document-level NMT models have been developed to address these issues, they face a significant data bottleneck. Most high-quality parallel corpora consist of isolated sentence pairs. Training context-aware models on a mixture of document-level data and context-less sentence pairs often leads the model to ignore context entirely or become confused by missing information.

Example of a sentence with its preceding context sentences in a document-level NMT setup.

Methods: Imputing Missing Context

The authors explore strategies to "fill in the blanks" when document-level context is unavailable in training data. They evaluate three primary imputation strategies:

  1. Random Context: Appending random, unrelated sentence pairs as context. This serves as a baseline but risks training the model to treat context as irrelevant noise.
  2. Partial Copy Heuristic: A simple yet effective approach where the model creates context by copying the source-target pair currently being translated, occasionally mixed with random sentences. This encourages the model to utilize the context window for lexical cues.
  3. Context Generation: A more complex method using a language model to generate plausible preceding sentences in the target language, which are then back-translated to create source-side context.

The underlying architecture is a standard Transformer, modified to accept concatenated sequences of multiple sentences as input.

Datasets and Evaluation

The models were trained on the OpenSubtitles2018 English-Russian dataset (~6M sentence pairs), where only 1.5M pairs have natural document-level context. Evaluation goes beyond standard BLEU scores, utilizing four "challenge sets" designed to measure specific linguistic phenomena:

  • Lexical Cohesion: Consistency of named entities.
  • Deixis: Correct use of politeness markers.
  • Verb Phrase Ellipsis: Correct translation of omitted verbs.
  • Morphological Inflection: Correct grammatical casing based on context.

Key Insights

  • BLEU vs. Coherence: Models with similar BLEU scores showed vastly different performance on challenge sets, indicating that BLEU is insufficient for evaluating document-level coherence.
  • The Danger of Randomness: Using purely random context degrades performance on long-distance linguistic tasks by teaching the model to "gate out" the context window.
  • The Copying Advantage: The partial copy heuristic was particularly effective for lexical cohesion. By seeing repeated words in the context, the model learns a strong prior for consistency, effectively acting as a "cache" for terminology.

Key Results

The "Partial Copy" model demonstrated significant improvements over the baseline and other imputation methods across all challenge sets.

Performance of different models on the four challenge sets (Deixis, Lexical Cohesion, Ellipsis). The Partial Copy model shows marked improvements, especially in lexical cohesion.

Specifically, on the En→Ru lexical cohesion task, the baseline scored 45.9%, while the "Partial Copy" model reached 74.9%. Furthermore, combining context imputation with tagged back-translation (using monolingual data to create synthetic documents) allowed the model to reach 90.1% accuracy on lexical cohesion and 89.6% on deixis.

Limitations and Future Work

While the copy heuristic is highly effective, it remains a heuristic that does not provide natural linguistic context. The Context Generation approach is more principled but currently limited by repetitive outputs and compounding errors from back-translation. Future research may focus on more sophisticated context "imagination" techniques and how these data augmentation schemes interact with specialized document-level architectures.

Summary updated

Abstract

Most neural machine translation systems still translate sentences in isolation. To make further progress, a promising line of research additionally considers the surrounding context in order to provide the model potentially missing source-side information, as well as to maintain a coherent output. One difficulty in training such larger-context (i.e. document-level) machine translation systems is that context may be missing from many parallel examples. To circumvent this issue, two-stage approaches, in which sentence-level translations are post-edited in context, have recently been proposed. In this paper, we instead consider the viability of filling in the missing context. In particular, we consider three distinct approaches to generate the missing context: using random contexts, applying a copy heuristic or generating it with a language model. In particular, the copy heuristic significantly helps with lexical coherence, while using completely random contexts hurts performance on many long-distance linguistic phenomena. We also validate the usefulness of tagged back-translation. In addition to improving BLEU scores as expected, using back-translated data helps larger-context machine translation systems to better capture long-range phenomena.
BibTeX
@misc{jean2019fill,
  author = {Sébastien Jean and Ankur Bapna and Orhan Firat},
  title = {{Fill in the Blanks: Imputing Missing Sentences for Larger-Context Neural Machine Translation}},
  year = {2019},
  eprint = {1910.14075},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.14075},
}
Figure