LIUM’s Contributions to the WMT2019 News Translation Task: Data and Systems for German-French Language Pairs

Fethi Bougares, Jane Wottawa, Anne Baillot, Loïc Barrault, Adrien BardetACL 2019

TL;DR

This paper introduces the German-French language pair to the WMT news translation task, addressing the over-reliance on English in machine translation research. The authors detail their data creation process and present Transformer-based neural machine translation systems that achieve competitive results, highlighting the importance of data quality and augmentation.

What Problem Does This Paper Address?

The field of machine translation research has historically been heavily skewed towards English as either a source or target language in shared tasks. This dominance is largely due to the abundance of English-language data. However, this reliance can mask crucial linguistic challenges, particularly those related to morphology and agreement, which are more prominent in languages like French and German. The authors argue that this imbalance limits the scope of research and can lead to systems that don't generalize well to languages with richer morphological structures. To address this, they introduce the German-French language pair into the WMT news translation task, aiming to foster research on these less-resourced directions.

Methods

The core of the LIUM submission is the Transformer architecture, a state-of-the-art sequence-to-sequence model that utilizes self-attention mechanisms. The authors employed both "small" and "big" configurations of the Transformer.

Key methodological aspects include:

  • Data Augmentation: To create sufficient parallel data for the German-French pair, they cross-matched existing EN-FR and EN-DE corpora from previous WMT tasks. Additionally, they utilized back-translation, a technique where monolingual data is translated into the target language using a preliminary NMT model, thereby generating synthetic parallel data.
  • Data Cleaning: A crucial step involved a rigorous cleaning pipeline that removed sentences with high overlap rates between source and target, aiming to improve data quality, especially for noisy corpora like ParaCrawl.
  • Ensemble Decoding: For improved performance, the authors ensembled predictions from multiple trained checkpoints (2 and 5 checkpoints) of their Transformer models.
  • Byte Pair Encoding (BPE): A vocabulary of 35K units was created using BPE for joint source and target language tokenization.

Datasets

The paper details the creation and use of several datasets:

  • Training Data:
    • Cross-matched data from previous WMT EN-FR and EN-DE corpora.
  • ParaCrawl: A large, but potentially noisy, parallel corpus.
  • Common Crawl: Another large parallel corpus.
  • Europarl-v7: Parallel data from European Parliament proceedings.
  • dev08-14: A smaller, older parallel corpus.
  • Monolingual News Crawl data (2018): Used for back-translation.
  • Development and Test Data:
    • Created from online news websites in French and German.
  • The development set was a collaborative effort with the University of Le Mans, involving students learning CAT tools.
  • The test set included professionally translated sentences for German to French.
  • Both dev and test sets were made publicly available.

Key Insights and Ideas

  • The Importance of Non-English Pairs: Introducing language pairs like German-French is vital for a more comprehensive understanding of MT challenges, especially those related to morphology and agreement, which are less apparent in English.
  • Data Quality is Paramount: The cleaning process significantly impacted performance, particularly for noisy corpora like ParaCrawl, demonstrating that cleaner data can lead to better translation quality.
  • Back-Translation is Effective: Augmenting training data with back-translated sentences provided a consistent improvement in BLEU scores, highlighting its utility for low-resource scenarios.
  • Big vs. Small Transformers: While the "big" Transformer architecture generally outperformed the "small" one, the gains were sometimes marginal, suggesting that careful data preparation and augmentation can significantly boost smaller models.

Key Results

The paper reports BLEU scores for various configurations:

  • German to French (DE→FR):
    • The best submitted system with back-translation achieved a BLEU score of 34.6.
  • Without back-translation, the best system (big Transformer, ensemble of 5) reached 27.61.
  • The official evaluation score for the German-French task was 33.4.
  • French to German (FR→DE):
    • The best submitted system (big Transformer, ensemble of 5, with back-translation) achieved a BLEU score of 22.34.
  • Without back-translation, the best system (big Transformer, ensemble of 5) reached 21.94.
  • The official evaluation score for the French-German task was 21.08 (small model) and 21.18 (big model).

The results show that the big Transformer models consistently outperformed the small ones, especially when ensembled. The inclusion of ParaCrawl data had mixed effects, sometimes improving results for the big model but occasionally degrading performance for the small model.

Limitations and Future Work

  • ParaCrawl's Impact: The paper notes that the impact of the ParaCrawl corpus was not always positive, especially for the small Transformer model in the FR→DE direction, suggesting that its noisy nature might require more sophisticated filtering or handling.
  • Data Creation Effort: The creation of the German-French parallel data involved significant effort in cross-matching and cleaning existing corpora, highlighting the challenges in building resources for new language pairs.
  • Future Work: While not explicitly detailed, the success of back-translation suggests its continued use. Further exploration into more advanced data selection and cleaning techniques for noisy corpora would be beneficial. The authors also implicitly suggest the continued importance of expanding MT research beyond English-centric tasks.

Summary updated

BibTeX
@inproceedings{bougares2019liums,
  author = {Fethi Bougares and Jane Wottawa and Anne Baillot and Loïc Barrault and Adrien Bardet},
  title = {{LIUM’s Contributions to the WMT2019 News Translation Task: Data and Systems for German-French Language Pairs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure