TL;DR
This paper details RWTH Aachen University's submission to the WMT 2019 machine translation task, focusing on the German-English, Chinese-English, and Kazakh-English language pairs. They achieved competitive results by leveraging the Transformer architecture, with significant improvements in the low-resource Kazakh-English task through data filtering, preprocessing, and transfer learning.
What Problem Does This Paper Address?
The paper addresses the challenge of building high-performing neural machine translation (NMT) systems for diverse language pairs, particularly those with limited parallel data. Specifically, it tackles issues like data quality in well-resourced languages (German-English, Chinese-English) and the scarcity of data in low-resource settings (Kazakh-English). The authors aim to improve translation quality by exploring various data preprocessing techniques, model architectures, and training strategies.
Methods
The core of RWTH Aachen's systems is the Transformer architecture, a standard in modern NMT. Their innovations lie in the meticulous application of various techniques:
- Data Filtering and Preprocessing: They employed sophisticated methods to clean and enhance training data. This included removing low-quality sentence pairs, handling special characters, and using techniques like deduplication and back-translation to augment data. For Chinese, they removed spaces and used a dictionary for character conversion.
- Subword Tokenization: Two variants of Byte Pair Encoding (BPE) were explored: the original BPE and the Unigram Language Model (ULM) approach (SentencePiece). ULM-BPE showed promising results, especially for Chinese-English.
- Fine-tuning and Ensembling: Systems were fine-tuned on in-domain data with lower learning rates. Ensembling multiple checkpoints of trained models was used to boost performance, particularly for the German-English task.
- Transfer Learning: For the low-resource Kazakh-English task, they leveraged transfer learning from related languages (Russian) and utilized additional monolingual and bilingual data.
- Handling Long Sentences: A specific strategy was developed to split long sentences during translation to improve performance on challenging test sets.
Datasets
The paper utilized several standard datasets for WMT tasks:
- German-English (De→En): CommonCrawl, Europarl, NewsCommentary, Rapid, and ParaCrawl.
- Chinese-English (Zh→En): A large corpus of Chinese-English sentence pairs, along with Chinese and English monolingual data.
- Kazakh-English (Kk→En): Limited Kk-En parallel data, supplemented with Russian-English (Ru-En) and Kazakh-Russian (Kk-Ru) parallel data, and monolingual data.
No new benchmarks were introduced; the focus was on improving performance on existing WMT tasks.
Key Insights and Ideas
- Data quality is paramount: Even with powerful architectures like the Transformer, aggressive data filtering and careful preprocessing can yield significant improvements, especially when dealing with noisy real-world data.
- ULM-BPE is effective: The Unigram Language Model approach to subword tokenization proved beneficial, particularly for languages with complex character sets like Chinese.
- Transfer learning is crucial for low-resource languages: Leveraging related languages and their abundant data is a viable strategy to overcome data scarcity in tasks like Kazakh-English translation.
- Long sentence splitting matters: Addressing the challenge of translating very long sentences by splitting them can lead to noticeable BLEU score improvements.
Key Results
- German-English: The final ensemble system achieved 39.6% BLEU on newstest2019, performing on par with their previous year's submission.
- Chinese-English: The system showed a 1.3% BLEU improvement over their previous submission, largely attributed to sentence splitting.
- Kazakh-English: This task saw the most substantial gains, with an 11.1% BLEU improvement over their baseline, reaching 24.2% BLEU on newstest2019. This was achieved through a combination of data augmentation, transfer learning, and careful fine-tuning.
Limitations and Future Work
- German-English: Despite extensive experimentation, no tested method provided a significant improvement over the previous year's system, indicating a potential plateau for current techniques on this specific task and dataset.
- Kazakh-English: The authors mention that time constraints prevented them from fully exploring larger model variants for transfer learning, suggesting this could be a future avenue. They also note that their experimental setup for combining language models with translation models on the Kk-En task was flawed due to overfitting on the development set, leading to a decrease in test performance. This highlights the need for careful validation and unseen test data.
Summary updated