Diverse Pretrained Context Encodings Improve Document Translation

Domenic Donato, Lei Yu, Chris Dyer2021 · DOI

Abstract

Domenic Donato, Lei Yu, Chris Dyer. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

2021.

Introduction

Standard Machine Translation (MT) systems typically process text sentence-by-sentence. While effective for isolated phrases, this "context-blind" approach often fails at the document level. Without broader context, models struggle with pronominal coreference (e.g., determining if "it" refers to a masculine or feminine noun from a previous sentence), maintaining tense consistency across a paragraph, and making appropriate lexical choices for specific domains.

The primary obstacle to document-level translation is data scarcity. While parallel sentence pairs are abundant, large-scale parallel documents—where inter-sentence context is preserved—are much rarer. Learning complex document structures from scratch alongside the translation task often proves difficult with limited data.

Methods: The Multi-context Architecture

To address these challenges, the authors propose a "Multi-context" Transformer architecture based on the adapter pattern. Instead of retraining the entire model, they integrate new "Parallel Attention Stacks" into a standard pretrained Transformer.

The Multi-context Transformer architecture incorporates diverse pretrained signals via parallel attention stacks.

Key components of this approach include:

  • Diverse Pretrained Signals: The model leverages embeddings from BERT (capturing local, word-in-context semantics) and PEGASUS (pre-trained for summarization, making it adept at capturing long-range document structure).
  • Parallel Attention Stacks: Each contextual signal—such as previous/next source sentences or previous target translations—is processed by its own attention block. These blocks are stacked, and their outputs are averaged with the primary sentence representation.
  • Multi-Stage Training: A robust sentence-level model is trained first. Subsequently, its parameters are frozen while only the new context-aware attention parameters are trained. This prevents "catastrophic forgetting" of the core translation task.
  • Drop-branch Regularization: During training, context branches are randomly "dropped" to ensure the model remains robust and does not over-rely on any single, potentially noisy signal.

Key Results

The Multi-context model was evaluated on several benchmarks, including NIST Chinese–English, IWSLT’14 English–German, and WMT’14 English–German.

Performance comparison of the Multi-context model against various baselines across three major datasets.

The model achieved state-of-the-art results, notably on the NIST Chinese-English task where it reached a BLEU score of 51.07, a significant improvement over the baseline Transformer's 46.69 (+4.38 BLEU). Qualitative analysis confirmed that the model successfully resolved ambiguous pronouns and maintained consistent verb tenses where sentence-level models failed.

Insights and Discussion

The Value of Diversity and Source Context

A key finding is that using multiple types of context (BERT + PEGASUS) consistently outperforms single-signal approaches. BERT provides a "micro" view of nearby words, while PEGASUS offers a "macro" view of document flow.

Furthermore, the authors found that conditioning on source context (the original document) is significantly more beneficial than conditioning on target context (previously generated translations). Target context is susceptible to "cascading errors," where a single mistake propagates through subsequent sentences.

The Data Threshold

Document-level models require a "critical mass" of data. The researchers observed that their model needs approximately 300,000 contextualized examples before it begins to consistently outperform a strong sentence-level baseline. Below this threshold, the added complexity of the context blocks can actually degrade performance.

Limitations

Despite its performance gains, the Multi-context model has a higher computational cost. Incorporating multiple external models like BERT and PEGASUS increases the memory footprint and reduces inference speed by approximately 2.5x compared to a standard Transformer. Future work may focus on more efficient signal merging to mitigate this overhead.

Summary updated

BibTeX
@misc{donato2021diverse,
  author = {Domenic Donato and Lei Yu and Chris Dyer},
  title = {{Diverse Pretrained Context Encodings Improve Document Translation}},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.104},
}
Figure