Kyoto University Participation to the WMT 2019 News Shared Task

Fabien Cromieres, Sadao KurohashiACL 2019

TL;DR

This paper details Kyoto University's participation in the WMT 2019 German-to-French news translation task. They achieved significant improvements by focusing on data cleaning, leveraging recent monolingual data through back-translation, and introducing a novel "checkpoint agreement" method to combine multiple trained models.

What Problem Does This Paper Address?

Machine translation (MT) systems, particularly Neural Machine Translation (NMT), often struggle with translating recent terminology or proper nouns that haven't appeared in their training data. This is especially true for language pairs like German-French, which have richer morphology than more commonly studied pairs. The WMT 2019 shared task presented this specific challenge, and the authors aimed to build a robust NMT system that could handle these nuances effectively. They also sought to improve upon standard NMT training and combination techniques.

Methods

The core of their system is the Transformer sequence-to-sequence model, implemented using the Tensor2Tensor framework. Key methodological contributions include:

  • Data Cleaning and Preprocessing: They meticulously cleaned the training data, addressing encoding issues (using ftfy) and filtering out uncommon Unicode characters. A crucial step was identifying and removing non-French/German sentences using a heuristic based on characteristic words and characters.
  • Leveraging Monolingual Data: To combat the issue of translating recent terms, they employed back-translation. They trained a French-to-German model and used it to translate recent monolingual German news data into French, effectively augmenting their training set with contemporary language.
  • Checkpoint Averaging: As a standard technique to improve model robustness, they averaged the weights of multiple checkpoints from a single training run.
  • Checkpoint Agreement for Model Combination: This is their novel contribution. Instead of traditional ensemble methods that require compatible model outputs, they proposed "checkpoint agreement." This method uses multiple checkpoints from a single model to estimate the confidence in a translation. If many checkpoints agree on a translation, it's considered more reliable. This confidence score can then be used to combine outputs from different models, even if they were trained with different settings (like subword vocabularies).

Datasets

  • Bilingual Data:
    • Europarl (approx. 1.7M sentence pairs)
    • Common Crawl (approx. 620k sentence pairs)
    • News Commentary (approx. 255k sentence pairs)
    • Note: Paracrawl corpus was not used.
  • Monolingual Data:
    • 2018 set of the News Crawl corpus (approx. 8M sentences) - used for back-translation.

Key Insights and Ideas

  • Data quality is paramount: Meticulous data cleaning, especially for encoding and language identification, significantly impacts translation quality.
  • Recent data is crucial: Monolingual data from recent news sources is vital for translating contemporary terms and events, as older bilingual corpora may not contain them. Back-translation is an effective way to leverage this.
  • Checkpoint agreement offers a model-agnostic confidence measure: The idea that multiple checkpoints from a single training run can signal translation confidence is powerful. This allows for a more flexible and implementation-independent way to combine diverse models.
  • Subword vocabulary size and word order don't significantly impact German-French: Unlike some language pairs, they found that varying subword vocabulary size or reversing French word order did not yield substantial improvements for this specific language pair.

Key Results

  • Baseline (bilingual data only): 33.18 cased BLEU.
  • Baseline + 2018 news data (monolingual): 35.92 cased BLEU (an improvement of +2.74). This highlights the impact of recent monolingual data.
  • Checkpoint averaging: 36.12 cased BLEU (+0.2 improvement over the previous step).
  • Selection through checkpoint agreement (single model): 36.23 cased BLEU (+0.3 improvement).
  • All Models combined with Checkpoint agreement: 36.73 cased BLEU (+0.81 improvement over the "Baseline + 2018 news data" result). This is their best result, showcasing the effectiveness of combining multiple models using their novel method.

Limitations and Future Work

  • Limitations:
    • The authors acknowledge that their "checkpoint agreement" method for combining models is a simple "majority vote" weighted by confidence. They suggest more complex combination algorithms (like those by Freitag et al., 2014) could potentially yield even better results.
  • They trained several models with different subword vocabularies and word orders but could not combine them using classic ensemble methods due to incompatible outputs. Their checkpoint agreement method was used to combine these models.
  • While they focused on 2018 news data for back-translation, they note that data from previous years might also have been useful.
  • Future Work:
    • Exploring more sophisticated system combination algorithms that integrate checkpoint agreement confidence scores.
  • Further investigation into the impact of subword segmentation and word order on different language pairs.

Summary updated

BibTeX
@inproceedings{cromieres2019kyoto,
  author = {Fabien Cromieres and Sadao Kurohashi},
  title = {{Kyoto University Participation to the WMT 2019 News Shared Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure