Document-Level Machine Translation with Large-Scale Public Parallel Corpora

Proyag Pal, Alexandra Birch, Kenneth HeafieldACL 2024 · OpenReview

Abstract

Despite the fact that document-level machine translation has inherent advantages over sentence-level machine translation due to additional information available to a model from document context, most translation systems continue to operate at a sentence level. This is primarily due to the severe lack of publicly available large-scale parallel corpora at the document level. We release a large-scale open parallel corpus with document context extracted from ParaCrawl in five language pairs, along with code to compile document-level datasets for any language pair supported by ParaCrawl. We train context-aware models on these datasets and find improvements in terms of overall translation quality and targeted document-level phenomena. We also analyse how much long-range information is useful to model some of these discourse phenomena and find models are able to utilise context from several preceding sentences.

Introduction

Most contemporary machine translation (MT) systems operate on a sentence-by-byte basis. While effective for isolated statements, this approach often fails when translation depends on preceding context. For instance, translating the English pronoun "it" into German requires knowing the grammatical gender of its referent in a previous sentence. The primary bottleneck for document-level MT (DocMT) has not been architectural, but rather a severe lack of large-scale, publicly available document-level parallel data. Most massive corpora, such as ParaCrawl, are typically released as shuffled sentence pairs, stripping away vital document structure.

Methodology and Data Reconstruction

To address this data scarcity, researchers developed a pipeline to reconstruct document context from the ParaCrawl corpus. By matching officially released sentence pairs against the raw, base64-encoded webpage text provided by ParaCrawl using source URLs, they recovered the original document structure.

The resulting dataset, ParaCrawl-Context, provides document-level context for five language pairs: English paired with German (deu), French (fra), Czech (ces), Polish (pol), and Russian (rus). For every sentence, up to 512 tokens of preceding text were extracted as context.

The models utilize a dual-encoder Transformer architecture. One encoder processes the current source sentence, while a secondary encoder handles context (either source or target side). These streams are integrated into the decoder via stacked cross-attention layers. To ensure robustness, the models were trained using variable-length context sampling.

Analysis of Contextual Information

A key finding is that target-side context (previously translated sentences) is generally more effective for improving translation quality than source-side context. However, this comes with a caveat known as "exposure bias." While using "Gold" (human-translated) context yields substantial gains, using "Predicted" context (the model's own previous outputs) is less effective due to error propagation.

Translation quality (BLEU / COMET) across different context configurations and language pairs.

Furthermore, the research explored the "diminishing returns" of historical context. While models can utilize information from several sentences back, the most significant improvements often come from the immediately preceding sentence.

Key Results

The context-aware models demonstrated significant improvements in handling specific discourse phenomena:

  • Pronoun Disambiguation: On the ContraPro (English→German) test set, a standard sentence-level model achieves roughly 50.7% accuracy. In contrast, the context-aware model reaches 78.5% accuracy with five sentences of context.
  • General Translation Quality: As shown in the evaluation across multiple language pairs, providing gold target context consistently improves BLEU and COMET scores.
  • Discourse Phenomena: Benchmarks like GTWiC showed steady improvements in handling deictic expressions as context length increased.
ContraPro accuracy (eng→deu) as a function of the number of sentences of target context.

Conclusion and Future Directions

While ParaCrawl-Context provides a massive leap forward, challenges remain. Web-crawled context can be "noisy," containing UI elements or boilerplate text rather than clean linguistic flow. Future work should focus on filtering for higher-quality context and bridging the performance gap between gold and predicted context through techniques like scheduled sampling. Additionally, expanding these datasets beyond English-centric pairs is vital for broader applicability in the MT community.

Summary updated

Abstract

Despite the fact that document-level machine translation has inherent advantages over sentence-level machine translation due to additional information available to a model from document context, most translation systems continue to operate at a sentence level. This is primarily due to the severe lack of publicly available large-scale parallel corpora at the document level. We release a large-scale open parallel corpus with document context extracted from ParaCrawl in five language pairs, along with code to compile document-level datasets for any language pair supported by ParaCrawl. We train context-aware models on these datasets and find improvements in terms of overall translation quality and targeted document-level phenomena. We also analyse how much long-range information is useful to model some of these discourse phenomena and find models are able to utilise context from several preceding sentences.
BibTeX
@inproceedings{pal2024document,
  author = {Proyag Pal and Alexandra Birch and Kenneth Heafield},
  title = {{Document-Level Machine Translation with Large-Scale Public Parallel Corpora}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=g4OIA8PFK3},
}
Figure