TL;DR
This paper presents the Charles University's submission to the WMT 2018 news translation task, focusing on improving low-resource language translation (English-Estonian) by leveraging related high-resource languages (English-Finnish) through a simple transfer learning approach. The key insight is that adapting a pre-trained "parent" model to a "child" low-resource language pair significantly boosts performance, even with limited child data.
What Problem Does This Paper Address?
Machine translation (MT) systems typically perform best when trained on large amounts of parallel data (sentences and their translations). However, for many language pairs, such as Estonian and English, this data is scarce, leading to lower translation quality. This paper tackles the challenge of improving translation for these low-resource languages by exploring how to effectively utilize data from related, higher-resource languages. The authors aim to bridge the performance gap between well-resourced and under-resourced language pairs.
Methods
The core of the proposed method is transfer learning, specifically a form of model adaptation. The process involves two main steps:
- Parent Model Training: A standard Neural Machine Translation (NMT) model, based on the Transformer architecture, is first trained on a high-resource language pair (e.g., English-Finnish).
- Child Model Adaptation: This pre-trained "parent" model is then further trained on the low-resource language pair (e.g., English-Estonian). Crucially, this adaptation is done without resetting the training hyperparameters or the adaptive learning rate. This "fine-tuning" approach allows the model to leverage the knowledge gained from the high-resource data.
A key requirement for this method is a shared vocabulary across both language pairs. This is achieved by learning wordpiece segmentations from the concatenated source and target sides of all available parallel data. To prevent bias towards the high-resource language, they employ a "balanced vocabulary" strategy, using only as many sentence pairs from the high-resource data as are available for the low-resource pair.
For English-Czech translation, the authors used a hybrid system from the previous year, combining phrase-based, transfer-based, and NMT approaches, primarily for comparison.
Datasets
The paper utilizes several datasets for training and evaluation:
- Parallel Corpora:
- English-Estonian: Europarl and Rapid corpora (approx. 0.8M sentences).
- English-Finnish: Standard WMT News data (approx. 2.8M sentences).
- English-Czech: Datasets described in Sudarikov et al. (2017), including back-translated Czech monolingual data (approx. 71.7M sentences).
- Monolingual Corpora:
- Estonian: News Crawl data (approx. 2.6M sentences).
- Finnish: News Crawl data (approx. 12.0M sentences).
- Czech: News Crawl data (approx. 59.2M sentences).
- Development Sets: WMT News 2018 for English-Estonian and WMT News 2015 for English-Finnish.
The authors also mention experiments with backtranslated data, where monolingual data from the target language is translated back into the source language to augment training sets.
Key Insights and Ideas
- Transfer Learning is Highly Effective for Low-Resource MT: The core finding is that adapting a model trained on a related high-resource language pair provides a substantial performance boost for low-resource pairs, even with limited target data.
- "Parent-Child" Model Adaptation Works: The simple fine-tuning approach, without resetting hyperparameters, is shown to be a robust method for knowledge transfer.
- Shared Vocabulary is Crucial: The success of the transfer learning hinges on having a common vocabulary representation across languages, achieved through wordpiece segmentation.
- Relatedness Matters: The effectiveness of transfer learning is implicitly linked to the linguistic relatedness between the high-resource and low-resource languages (Finnish and Estonian are closely related).
- Backtranslation Further Improves Performance: While not used in the submitted models, experiments showed that backtranslated data can further enhance translation quality, especially when combined with transfer learning.
- Even Small Amounts of Child Data Benefit: The experiments simulating low-resource scenarios by reducing child data showed that the relative improvement from transfer learning increases as child data decreases, highlighting its utility for extremely data-scarce situations.
Key Results
- English-Estonian: The transfer learning approach yielded a significant performance boost. For example, the "Transfer" model achieved a BLEU score of 19.74 for EN-ET, compared to a baseline of 17.03 and "Only Parent" (no adaptation) of 2.32. Similar improvements were observed for ET-EN.
- English-Finnish: The improvement from transfer learning was smaller or even negligible, as Finnish is a higher-resource language compared to Estonian.
- English-Czech: The hybrid system was used for comparison, showing a widening gap between hybrid and purely neural systems over previous years.
- Effect of Child Data Size: When reducing English-Estonian training data from 800k to 10k sentence pairs, the relative improvement from transfer learning increased significantly, demonstrating its value for extremely low-resource settings.
Limitations and Future Work
- Backtranslation Not Used in Submission: The submitted models for Estonian and Finnish did not include the backtranslation step, which was shown to provide further gains. This was likely due to time constraints or submission requirements.
- "Balanced Vocabulary" Strategy: The paper mentions not experimenting with other proportions of data for the balanced vocabulary, suggesting this could be an area for future exploration.
- Comparison to Hybrid Systems: The English-Czech submission was primarily for comparison, highlighting the continued dominance of neural systems over hybrid approaches.
- Future Work: The authors implicitly suggest that exploring more sophisticated transfer learning techniques, as well as further leveraging backtranslation, could lead to even better results. The paper also points to the general trend of neural systems outperforming hybrid systems, indicating a future shift towards purely neural architectures.
Summary updated