The RWTH Aachen University Machine Translation Systems for WMT 2019

Jan Rosendahl, Christian Herold, Yunsu Kim, Miguel Graça, Weiyue Wang, Parnia Bahar, Yingbo Gao, Hermann NeyACL 2019

TL;DR

This paper details RWTH Aachen University's submission to the WMT 2019 machine translation task, focusing on the German-English, Chinese-English, and Kazakh-English language pairs. They achieved competitive results by leveraging the Transformer architecture, with significant improvements in the low-resource Kazakh-English task through data filtering, preprocessing, and transfer learning.

What Problem Does This Paper Address?

The paper addresses the challenge of building high-performing neural machine translation (NMT) systems for diverse language pairs, particularly those with limited parallel data. Specifically, it tackles issues like data quality in well-resourced languages (German-English, Chinese-English) and the scarcity of data in low-resource settings (Kazakh-English). The authors aim to improve translation quality by exploring various data preprocessing techniques, model architectures, and training strategies.

Methods

The core of RWTH Aachen's systems is the Transformer architecture, a standard in modern NMT. Their innovations lie in the meticulous application of various techniques:

  • Data Filtering and Preprocessing: They employed sophisticated methods to clean and enhance training data. This included removing low-quality sentence pairs, handling special characters, and using techniques like deduplication and back-translation to augment data. For Chinese, they removed spaces and used a dictionary for character conversion.
  • Subword Tokenization: Two variants of Byte Pair Encoding (BPE) were explored: the original BPE and the Unigram Language Model (ULM) approach (SentencePiece). ULM-BPE showed promising results, especially for Chinese-English.
  • Fine-tuning and Ensembling: Systems were fine-tuned on in-domain data with lower learning rates. Ensembling multiple checkpoints of trained models was used to boost performance, particularly for the German-English task.
  • Transfer Learning: For the low-resource Kazakh-English task, they leveraged transfer learning from related languages (Russian) and utilized additional monolingual and bilingual data.
  • Handling Long Sentences: A specific strategy was developed to split long sentences during translation to improve performance on challenging test sets.

Datasets

The paper utilized several standard datasets for WMT tasks:

  • German-English (De→En): CommonCrawl, Europarl, NewsCommentary, Rapid, and ParaCrawl.
  • Chinese-English (Zh→En): A large corpus of Chinese-English sentence pairs, along with Chinese and English monolingual data.
  • Kazakh-English (Kk→En): Limited Kk-En parallel data, supplemented with Russian-English (Ru-En) and Kazakh-Russian (Kk-Ru) parallel data, and monolingual data.

No new benchmarks were introduced; the focus was on improving performance on existing WMT tasks.

Key Insights and Ideas

  • Data quality is paramount: Even with powerful architectures like the Transformer, aggressive data filtering and careful preprocessing can yield significant improvements, especially when dealing with noisy real-world data.
  • ULM-BPE is effective: The Unigram Language Model approach to subword tokenization proved beneficial, particularly for languages with complex character sets like Chinese.
  • Transfer learning is crucial for low-resource languages: Leveraging related languages and their abundant data is a viable strategy to overcome data scarcity in tasks like Kazakh-English translation.
  • Long sentence splitting matters: Addressing the challenge of translating very long sentences by splitting them can lead to noticeable BLEU score improvements.

Key Results

  • German-English: The final ensemble system achieved 39.6% BLEU on newstest2019, performing on par with their previous year's submission.
  • Chinese-English: The system showed a 1.3% BLEU improvement over their previous submission, largely attributed to sentence splitting.
  • Kazakh-English: This task saw the most substantial gains, with an 11.1% BLEU improvement over their baseline, reaching 24.2% BLEU on newstest2019. This was achieved through a combination of data augmentation, transfer learning, and careful fine-tuning.

Limitations and Future Work

  • German-English: Despite extensive experimentation, no tested method provided a significant improvement over the previous year's system, indicating a potential plateau for current techniques on this specific task and dataset.
  • Kazakh-English: The authors mention that time constraints prevented them from fully exploring larger model variants for transfer learning, suggesting this could be a future avenue. They also note that their experimental setup for combining language models with translation models on the Kk-En task was flawed due to overfitting on the development set, leading to a decrease in test performance. This highlights the need for careful validation and unseen test data.

Summary updated

BibTeX
@inproceedings{rosendahl2019rwth,
  author = {Jan Rosendahl and Christian Herold and Yunsu Kim and Miguel Graça and Weiyue Wang and Parnia Bahar and Yingbo Gao and Hermann Ney},
  title = {{The RWTH Aachen University Machine Translation Systems for WMT 2019}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure