Context-Aware Monolingual Repair for Neural Machine Translation

Elena Voita, Rico Sennrich, Ivan Titov

Authors: Elena Voita, Rico Sennrich, Ivan Titov.
Source: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Neural Machine Translation (NMT) has reached impressive levels of fluency at the sentence level, yet it often falls into an "uncanny valley" when translating entire documents. Because most NMT systems process sentences in isolation, they remain "context-blind," leading to disjointed texts where pronouns, genders, and terminology fluctuate inconsistently. This paper introduces DocRepair, a monolingual post-editing model designed to harmonize document translations without the need for scarce document-level parallel corpora.

The Challenge of Document-Level Translation

Standard NMT models are typically trained on sentence pairs. While effective for short snippets, this approach fails to maintain discourse-level coherence. For instance, a character introduced with a masculine pronoun in one sentence might be referred to with a feminine one in the next, or a specific technical term might be translated using different synonyms throughout a chapter.

While researchers have attempted to build context-aware models, these usually require document-level parallel data—source and target documents paired together. Such data is significantly harder to acquire than simple sentence pairs or raw monolingual text in the target language. DocRepair addresses this bottleneck by leveraging target-language monolingual data to learn discourse patterns.

The DocRepair Framework

The authors propose a two-step pipeline. First, a standard NMT model translates sentences in isolation. Second, the DocRepair model—a monolingual sequence-to-sequence Transformer—takes a block of these translations and "repairs" them to ensure consistency.

The DocRepair training pipeline: generating synthetic inconsistencies through round-trip translation.

The core innovation lies in the training process, which utilizes synthetic inconsistency. To train the model without parallel data, the authors follow these steps:

  1. Consistent Data Selection: They identify high-quality monolingual text blocks in the target language (e.g., Russian).
  2. Synthetic Error Generation: Each sentence in the block is passed through a "round-trip" translation (Target $\rightarrow$ Source $\rightarrow$ Target) in isolation. Because this back-and-forth process occurs sentence-by-sentence, the resulting block loses its original contextual harmony.
  3. The Repair Task: DocRepair is trained to map these inconsistent "round-trip" blocks back to the original, consistent monolingual text.

This approach effectively uses round-trip translation as a noise source that mimics the exact distribution of errors a sentence-level MT system makes.

Results and Discourse Consistency

Evaluated on English-Russian translations from the OpenSubtitles2018 dataset, DocRepair demonstrated significant improvements in both general quality and specific discourse phenomena. The model improved the BLEU score from 33.91 to 34.60, outperforming more complex models that rely on parallel document data.

Consistency scores for various discourse phenomena as a function of training steps.

The gains in targeted consistency benchmarks were particularly striking:

  • Deixis (T-V distinction): Accuracy jumped from 50.0% to 91.8%.
  • Lexical Cohesion: Accuracy increased from 45.9% to 80.6%.

Human evaluations further supported these metrics; in a head-to-head comparison of 700 examples, raters preferred the DocRepair version over the baseline in 73% of cases where the model made a change.

Discussion and Limitations

One notable exception to the model's success was Verb Phrase (VP) ellipsis, where performance actually declined. The authors suggest that round-trip translation fails to accurately simulate how English VP ellipsis (e.g., "he did") is mistranslated into Russian, indicating that some phenomena may still require source-side context to resolve correctly.

Additionally, while the current study focused on 4-sentence fragments, scaling this to longer documents remains a challenge for Transformer memory. However, the success of DocRepair proves that many aspects of document "flow" can be learned purely from target-language patterns, allowing for a "black-box" compatible post-processor that can improve any existing NMT system without retraining.

Summary updated

BibTeX
@inproceedings{voita2019context,
  author = {Elena Voita and Rico Sennrich and Ivan Titov},
  title = {{Context-Aware Monolingual Repair for Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=s5wx3YWt-U_},
}
Figure