TL;DR
The RWTH Aachen team achieved top performance in the WMT 2018 news translation tasks, particularly for German-English, by leveraging ensembles of Transformer-based neural machine translation systems. They meticulously explored data selection, fine-tuning strategies, and model configurations, demonstrating significant improvements over previous years and winning systems.
What Problem Does This Paper Address?
The paper addresses the challenge of building high-performing machine translation (MT) systems for real-world news text. Specifically, it focuses on the EMNLP 2018 Third Conference on Machine Translation (WMT 2018) shared tasks, which involve translating between German-English, English-Turkish, and Chinese-English. The core problem is to push the state-of-the-art in MT quality, measured by metrics like BLEU, TER, and CTER, by optimizing neural MT architectures and training methodologies. This includes dealing with varying data availability (low-resource vs. high-resource languages) and the complexities of large-scale training.
Methods
The RWTH Aachen team primarily employed the Transformer architecture, a now-standard but highly effective neural MT model, implemented using the Sockeye toolkit. Their approach involved several key methodological choices:
- Ensembling: Combining multiple Transformer models to achieve better generalization and robustness.
- Data Preprocessing: A pipeline involving text normalization, tokenization (Moses), frequent casing (Jane toolkit), and Chinese segmentation (Jieba).
- Byte-Pair Encoding (BPE): Used to segment words into subword units, which is crucial for handling rare words and morphology. They experimented with different merge operation counts (20k and 50k), finding 50k to be superior.
- Data Selection and Filtering: Crucially, they explored the impact of including and filtering various data sources, notably the ParaCrawl corpus. Filtering ParaCrawl proved beneficial for German-English.
- Fine-tuning: Adapting models trained on large datasets to specific test sets or domains, often using older test sets to improve performance on newer ones.
- Hyperparameter Optimization: Extensive experimentation with parameters like batch size, model dimension, number of layers, attention heads, and learning rate schedules.
- Regularization Techniques: For low-resource settings (English-Turkish), they applied significant dropout and weight tying to prevent overfitting.
- Back-translation: Generating synthetic parallel data by translating monolingual data from the target to the source language using an existing MT system.
- Gradient Accumulation: A technique to simulate larger batch sizes when GPU memory is limited, which they found to be effective.
Datasets
The paper utilized a variety of datasets for training and evaluation across the different language pairs:
- German-English:
- Training: WMT 2018 standard parallel data (Europarl, CommonCrawl, NewsCommentary, Rapid), ParaCrawl (filtered), NewsCrawl 2015 & 2017, and synthetic data.
- Development/Test: newstest2015, newstest2017, newstest2018.
- English-Turkish:
- Training: A low-resource parallel dataset of approximately 200k sentences, supplemented with back-translated data from Turkish NewsCrawl.
- Development/Test: newstest2018.
- Chinese-English:
- Training: A large parallel dataset totaling 24.7M sentence pairs (620M English, 547M Chinese words), including monolingual Chinese NewsCrawl data for back-translation.
- Development/Test: newsdev2017, newstest2017, newstest2018.
Key Insights and Ideas
- Transformer Dominance: The Transformer architecture, even with standard configurations, provides a very strong baseline, significantly outperforming previous RNN-based systems.
- Data Filtering is Key: For German-English, filtering the large but noisy ParaCrawl corpus was essential for improving performance, highlighting the importance of data quality over sheer quantity.
- Model Size and Batch Size Matter: Larger models (increased
d_modelandd_ff) and larger effective batch sizes (achieved through multi-GPU training and gradient accumulation) consistently led to better results. - Regularization for Low-Resource: In low-resource scenarios like English-Turkish, aggressive regularization (dropout, weight tying) is critical to prevent overfitting and achieve competitive scores.
- Back-translation Benefits: Synthetic data generated via back-translation can significantly boost performance, especially when combined with fine-tuning.
- Ensembling for the Win: While single strong Transformer models performed well, ensembling multiple such models provided the final edge, leading to top scores.
- Fine-tuning on Old Data: Using older test sets for fine-tuning proved to be an effective strategy for adapting models to the news domain.
Key Results
- German-English: Achieved first place in WMT 2018, with their best system scoring 37.5% BLEU on newstest2015 and 48.4% BLEU on newstest2018. This represented a 6.8% BLEU improvement over their own WMT 2017 submission and a 4.8% BLEU improvement over the 2017 winning system.
- English-Turkish: Showed a 3.6% BLEU improvement over the previous year's winning system, reaching 18.0% BLEU on newstest2018 with their best ensemble system.
- Chinese-English: Improved by 2.2% BLEU on average over their baseline systems, though they finished behind the 2018 winning systems. Their best single system achieved 26.7% BLEU on newstest2018.
Limitations and Future Work
- Convergence Time: Due to time constraints and the scale of experiments, some reported models did not fully converge, suggesting potential for further gains.
- Hyperparameter Sensitivity: The paper highlights the importance of various hyperparameters, implying that further fine-tuning could yield additional improvements.
- Ensemble Complexity: While ensembles are effective, they increase computational cost for inference. Future work could explore more efficient ensembling or distillation techniques.
- Data Quality: The impact of data filtering (e.g., ParaCrawl) suggests that continued efforts in data curation and cleaning are vital for MT research.
- Low-Resource Challenges: Despite improvements, the English-Turkish task remains challenging due to limited parallel data, indicating a need for more advanced techniques for low-resource MT.
Summary updated