Context-Adaptive Document-Level Neural Machine Translation

Linlin Zhang, Zhirui Zhang, Boxing Chen, Weihua Luo, Luo SiICASSP 2022 · DOI · OpenReview

TL;DR

This paper tackles the challenge of document-level neural machine translation (DocMT) by proposing a novel "context-adaptive" approach. Instead of relying on a fixed context window, their method uses a lightweight predictor to dynamically select the most relevant source sentence context for each translation, significantly improving translation quality.

What Problem Does This Paper Address?

Standard neural machine translation (NMT) models translate sentences in isolation, ignoring the broader document context. While document-level NMT (DocMT) aims to address this by incorporating context from surrounding sentences, existing methods often use a fixed amount of context (e.g., the previous N sentences). This rigidity can be problematic: too much context might introduce noise, while too little might miss crucial discourse dependencies. The authors argue that the optimal context size varies per sentence and propose a data-adaptive method to select the most helpful context dynamically.

Methods

The core innovation is a lightweight context predictor integrated into existing DocMT frameworks. This predictor takes the encoder outputs of the current source sentence and, using a simple feed-forward network and softmax, outputs probabilities for different context options. These options include not using any context, using previous sentences, or using future sentences.

The predictor is trained jointly with the DocMT model. The training loss is a combination of:

  • Translation Loss (LMT): The standard NMT loss, weighted by the predictor's confidence in each context option.
  • Diversity Loss (Ldiv): A KL-divergence term encouraging the predictor to explore diverse context options.
  • Uniformity Loss (Luni): A loss that pushes the predictor towards making unambiguous selections during inference.
  • Masked Loss (Lmask): Inspired by BERT, this likely involves masked language modeling tasks on the source sentences.

During inference, the predictor selects the context option with the highest probability, and this selected context is then fed into the DocMT model. The authors apply this predictor to two baseline DocMT models: a "Context-Unit Model" and a "Concatenate Model."

Datasets

The experiments were conducted on four widely used document-level parallel datasets:

  • TED (ZH-EN/EN-DE): Derived from IWSLT 2015 and 2017 evaluation campaigns.
  • EN-DE: Three corpora extracted by Maruf et al. (2019).

Key Insights and Ideas

  • Context is not one-size-fits-all: The paper strongly supports the idea that the optimal amount of context for translation varies from sentence to sentence. A fixed context window is suboptimal.
  • Lightweight prediction is effective: The introduction of a simple, lightweight predictor significantly boosts performance without adding substantial computational overhead during inference.
  • Dynamic selection improves consistency: By adaptively choosing context, the model can better handle phenomena like pronoun resolution and tense consistency, leading to more natural translations. The example in Table 5 shows how the model can correctly pick up the tense from a following sentence, something a fixed previous-context model might miss.

Key Results

The proposed context-adaptive DocMT approach achieved significant improvements over previous baselines.

  • On the concatenate model, the method gained up to 1.99 BLEU points over the baseline across various datasets (TED ZH-EN, TED EN-DE, News, and Europarl).
  • The context-unit model also saw significant gains.
  • Inference time increase is minimal: Despite the added predictor, the decoding time per batch increased only slightly, indicating the predictor is indeed lightweight.
  • Ablation studies confirmed the importance of the predictor and its associated losses (Luni, Ldiv) for achieving the performance gains.

Limitations and Future Work

  • Training Time: While inference is efficient, training the model with N context options requires training the model N times, increasing overall training time per epoch. However, the authors note that the number of convergence rounds might be reduced.
  • Parameter Count: The paper states that the model parameters increase "very little," implying that the predictor adds only a small number of parameters compared to the DocMT model itself.
  • Future Work: Not explicitly detailed, but the success of adaptive context selection suggests further exploration into more sophisticated context selection mechanisms or applying this adaptive approach to other document-level NLP tasks.

Summary updated

Abstract

Document-level translation models are still far from perfect. Most existing document-level neural machine translation (NMT) models leverage a fixed number of the previous or all global sentences to handle the context-independent problem in standard NMT. However, the translating of each source sentence benefits from various sizes of context. And study shows that inappropriate redundant context will increase model burden but not improve the translation performance. This work introduces a data-adaptive method that enables the model to adopt the necessary and helpful context. Specifically, we introduce a light predictor into two document-level translation models to select the explicit context. Experiments demonstrate the proposed approach can significantly improve the performance over the previous methods with a gain up to 1.99 BLEU points.
BibTeX
@inproceedings{zhang2022context,
  author = {Linlin Zhang and Zhirui Zhang and Boxing Chen and Weihua Luo and Luo Si},
  title = {{Context-Adaptive Document-Level Neural Machine Translation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746245},
}
Figure