English-Czech Systems in WMT19: Document-Level Transformer

Martin Popel, Dominik Macháček, Michal Auersperger, Ondřej Bojar, Pavel Pecina

TL;DR

Researchers from Charles University improved English-Czech machine translation by moving beyond isolated sentences to document-level context. By training Transformer models on multi-sentence segments, they achieved a modest but statistically significant +0.6 BLEU improvement and demonstrated how broader context can resolve tricky lexical ambiguities that single-sentence models miss.

What Problem Does This Paper Address?

Standard Neural Machine Translation (NMT) typically treats a document as a bag of independent sentences. While this works surprisingly well, it ignores inter-sentential phenomena like pronoun resolution, consistent tense usage, and lexical coherence. For example, a polysemous English word like "goal" might be translated as a sports score in one sentence and a life ambition in the next, even if the document is clearly about football.

Human evaluators often find that while NMT systems approach "human parity" on isolated sentences, they fall significantly short when evaluated on whole documents. The authors aimed to bridge this gap for the WMT19 English-Czech news translation task by providing the model with surrounding context during both training and inference.

Methods

The authors built their systems using two popular NMT frameworks: Tensor2Tensor (T2T) and Marian. Their core innovation was Training Data Context Augmentation. Instead of feeding the model single sentence pairs, they extracted sequences of consecutive sentences up to a limit of 1000 characters (averaging about 15 sentences).

To handle these longer sequences during translation (decoding), they employed two different strategies:

  1. Sliding Window (Marian): They translated 3-sentence blocks and primarily used the translation of the middle sentence, which benefited from both preceding and succeeding context.
  2. Overlapping Splits (T2T): They defined "pre-context" and "post-context" sentences that were fed into the model to provide background but were discarded in the final output, keeping only the "main content" translations.

They also focused on reimplementing their successful WMT18 system in the Marian framework to compare performance across different engineering stacks.

Datasets

The models were trained on a massive collection of parallel and synthetic data allowed for the WMT19 shared task, totaling approximately 184 million sentence pairs.

Training data sizes (in thousands)

Key sources included:

  • CzEng 1.7: The primary parallel corpus.
  • Europarl, News Commentary, CommonCrawl, and WikiTitles.
  • Backtranslated data: Large amounts of synthetic data created by translating Czech news mono-data back into English.

Key Insights and Ideas

  • Context Resolves Ambiguity: The most significant "aha" moment was seeing the document-level model correctly choose between non-synonymous Czech lemmata. For instance, the word "goal" was correctly translated as gól (sports) rather than cíl (aim) when the surrounding sentences mentioned sports context.
  • Framework Parity is Hard: Despite using identical data and similar hyperparameters, the T2T implementation outperformed the Marian implementation by about 1.7 BLEU points. The authors hypothesize this is due to subtle differences in optimizers (Adafactor vs. Adam) and batching strategies.
  • The "Weak Evidence" Challenge: While document-level models feel intuitively better, proving it quantitatively is difficult. The authors found that cases where inter-sentential context is absolutely necessary for correct translation are relatively rare in standard news datasets, making it hard to draw sweeping conclusions from BLEU scores alone.

Key Results

The DocTransformer T2T system emerged as the top performer among their submissions.

Automatic evaluation results on newstest2019
  • BLEU Gains: The document-level T2T system achieved 31.03 BLEU (uncased), a +0.6 improvement over applying the same model to isolated sentences.
  • Qualitative Success: Manual analysis confirmed that the document-level models corrected specific lexical errors. As shown in the appendix, the document-level models were better at identifying that "running" referred to a political candidacy rather than physical movement, and "settlement" referred to a legal compensation rather than a village.
Examples of errors corrected by document-level models

Limitations and Future Work

The authors noted several limitations:

  • Efficiency: Decoding with large contexts is computationally expensive and time-consuming.
  • Sampling Bias: Their context augmentation method implicitly upsamples sentences from longer documents, an effect they leave for future analysis.
  • Evaluation Metrics: Standard metrics like BLEU are not sensitive enough to capture the subtle improvements in document coherence.

Future work involves exploring more sophisticated sampling methods and finding better ways to evaluate inter-sentential consistency beyond manual spot-checks.

Summary updated

Abstract

We describe our NMT systems submitted to the WMT19 shared task in English-Czech news translation. Our systems are based on the Transformer model implemented in either Tensor2Tensor (T2T) or Marian framework. We aimed at improving the adequacy and coherence of translated documents by enlarging the context of the source and target. Instead of translating each sentence independently, we split the document into possibly overlapping multi-sentence segments. In case of the T2T implementation, this "document-level"-trained system achieves a $+0.6$ BLEU improvement ($p<0.05$) relative to the same system applied on isolated sentences. To assess the potential effect document-level models might have on lexical coherence, we performed a semi-automatic analysis, which revealed only a few sentences improved in this aspect. Thus, we cannot draw any conclusions from this weak evidence.
BibTeX
@inproceedings{popel2019english,
  author = {Martin Popel and Dominik Macháček and Michal Auersperger and Ondřej Bojar and Pavel Pecina},
  title = {{English-Czech Systems in WMT19: Document-Level Transformer}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure