English-Czech Systems in WMT19: Document-Level Transformer

Martin Popel, Dominik Macháček, Michal Auersperger, Ondřej Bojar, Pavel Pecina

TL;DR

This paper presents NMT systems for English-Czech translation that go beyond sentence-by-sentence processing by considering multi-sentence segments. While their document-level Transformer model showed a slight BLEU improvement, the impact on lexical consistency was minimal, suggesting that context-aware translation still faces challenges in capturing subtle inter-sentential phenomena.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) has achieved remarkable quality, often matching or even surpassing human translation on isolated sentences. However, evaluating translation quality solely on individual sentences can be misleading. Systems that ignore document-level context might not be penalized for errors that become apparent only when considering the broader discourse, while systems that do leverage context might be unfairly penalized for deviations that are acceptable within a larger text. This paper tackles the challenge of improving NMT by incorporating document-level context, aiming to enhance both overall translation quality and lexical consistency across sentences.

Methods

The authors developed five NMT systems for the WMT19 English-Czech news translation task. Their core innovation lies in adapting the Transformer architecture to process multi-sentence segments instead of single sentences. This was achieved by:

  • Context Augmentation: Training data was augmented by creating overlapping multi-sentence sequences. These sequences, containing up to 1000 characters, were fed to the Transformer to allow it to learn inter-sentence dependencies.
  • Two Frameworks: Systems were implemented using either the Tensor2Tensor (T2T) or the Marian framework.
  • Document-Level Decoding: Novel decoding strategies were employed to leverage the document-level context during translation. For Marian, this involved considering up to three preceding sentences. For T2T, they experimented with both non-overlapping and overlapping multi-sentence splits, with overlapping splits showing better results.
  • Lexical Consistency Analysis: A semi-automatic analysis was performed to assess if document-level models improved the consistent translation of polysemous words within a document.

Datasets

The training data for the WMT19 shared task was used, constrained to what was allowed for the competition. This included standard parallel corpora like CzEng 1.7, Europarl v7, News Commentary v12, Common Crawl, and WikiTitles, along with synthetic data from backtranslation (CS NewsCrawl 2016-17, CS NewsCrawl 2007-17, CS NewsCrawl 2018). The test set was newstest2019. No new benchmarks were introduced; the focus was on applying document-level techniques to existing datasets.

Key Insights and Ideas

  • Document Context is Tricky: While the idea of using document context for NMT is intuitively appealing for better coherence, the paper demonstrates that achieving significant improvements, especially in lexical consistency, is not straightforward.
  • Slight BLEU Gains, but Limited Lexical Impact: The document-level T2T system showed a modest BLEU improvement over its sentence-level counterpart. However, the manual analysis for lexical consistency revealed very few instances where the document-level model performed better, suggesting that inter-sentential phenomena requiring context for disambiguation are relatively rare.
  • Decoding Strategies Matter: The choice of decoding strategy for document-level models significantly impacts performance. The authors found that a 3-sentence context for Marian and overlapping splits for T2T yielded the best results.
  • Framework Differences: The paper highlights differences in performance and training between the T2T and Marian frameworks, with T2T generally outperforming Marian in BLEU scores for comparable systems. This is attributed to factors like optimizer choice (Adam vs. Adafactor), GPU resources, and checkpoint averaging methods.

Key Results

  • The "DocTransformer T2T" system achieved a +0.6 BLEU improvement (p < 0.05) compared to the same system applied on isolated sentences.
  • However, the semi-automatic analysis for lexical consistency found only one (for Marian) and three (for T2T) instances where the document-level variant performed better than the non-context variant.
  • In the automatic evaluation on newstest2019, the "DocTransformer T2T" system scored 31.03 BLEU (uncased), while the "DocTransformer Marian" scored 29.17 BLEU (uncased). The best performing system overall was the UEdin system with 31.83 BLEU.
  • The authors noted that applying their document-level T2T system on isolated sentences led to a significant worsening (-0.6 BLEU).

Limitations and Future Work

The authors acknowledge that the manual evaluation for lexical consistency was based on a very small number of examples. They conclude that more data is needed to draw definitive conclusions about the benefits of document-level models for lexical consistency. They also suggest that the observed rarity of cases where inter-sentential context is crucial for disambiguating polysemous words might explain the limited gains. Future work could explore more sophisticated context augmentation techniques or investigate different types of document-level phenomena beyond lexical consistency.

Summary updated

Abstract

We describe our NMT systems submitted to the WMT19 shared task in English-Czech news translation. Our systems are based on the Transformer model implemented in either Tensor2Tensor (T2T) or Marian framework. We aimed at improving the adequacy and coherence of translated documents by enlarging the context of the source and target. Instead of translating each sentence independently, we split the document into possibly overlapping multi-sentence segments. In case of the T2T implementation, this "document-level"-trained system achieves a $+0.6$ BLEU improvement ($p<0.05$) relative to the same system applied on isolated sentences. To assess the potential effect document-level models might have on lexical coherence, we performed a semi-automatic analysis, which revealed only a few sentences improved in this aspect. Thus, we cannot draw any conclusions from this weak evidence.
BibTeX
@inproceedings{popel2019english,
  author = {Martin Popel and Dominik Macháček and Michal Auersperger and Ondřej Bojar and Pavel Pecina},
  title = {{English-Czech Systems in WMT19: Document-Level Transformer}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure