TL;DR
This paper details the Universitat d'Alacant's approach to the English-to-Kazakh news translation task at WMT 2019. They achieved a strong 2nd place ranking in chrF++ by leveraging iterative back-translation, transfer learning from related languages, and a novel hybrid system combining neural machine translation (NMT) with a rule-based system.
What Problem Does This Paper Address?
The core challenge tackled is the English-to-Kazakh language pair's low-resource nature. This means a scarcity of high-quality parallel data (sentences aligned in both languages), which is crucial for training effective NMT systems. Traditional NMT models struggle with such limited data, leading to suboptimal translation quality. The authors aim to overcome this data bottleneck by creatively utilizing available resources, including monolingual data and parallel data from other language pairs.
Methods
The Universitat d'Alacant team employed a multi-pronged strategy:
- Iterative Back-translation: This technique is used to generate synthetic parallel data from monolingual corpora. The process involves repeatedly translating monolingual data into the target language using an NMT system, then using this synthetic data to retrain and improve the NMT system. This is done iteratively to progressively enhance translation quality.
- Transfer Learning & Pivot Back-translation: To leverage data from related languages, they used "pivot" languages (like Russian) to bridge English and Kazakh. This involves training models on English-Russian and Russian-Kazakh data, then using these to facilitate English-Kazakh translation. They also experimented with "transfer learning," where models pre-trained on high-resource language pairs are fine-tuned on the low-resource English-Kazakh data.
- Morphological Segmentation: For Kazakh, a highly inflected language, they applied morphological segmentation. This breaks down words into their constituent parts (stem and suffixes), which can help NMT systems generalize better to unseen word forms.
- Hybridization with Rule-Based Machine Translation (RBMT): They integrated the Apertium English-to-Kazakh RBMT system into their NMT pipeline. This "multi-source" approach allows the NMT system to draw knowledge from both the English input and the Apertium translation, potentially capturing linguistic nuances missed by NMT alone.
Datasets
The paper utilized several datasets:
- Parallel Corpora:
- English-Kazakh (en-kk): News, Commentary, and Wikititles.
- English-Russian (en-ru) and Kazakh-Russian (kk-ru) for transfer learning and pivot back-translation.
- Monolingual Corpora:
- Kazakh: News Crawl, Wiki dumps, Common Crawl.
- English: News Crawl (a subset was used).
- Development/Test Sets:
newsdev2019was split for development and testing during training, andnewstest2019was used for final automatic evaluation.
Key Insights and Ideas
- Iterative back-translation is powerful: The iterative nature of back-translation, when applied multiple times, significantly boosted translation quality, outperforming systems trained only on genuine parallel data.
- Morphological segmentation helps: For Kazakh, segmenting words into stems and suffixes improved generalization, especially for inflected forms, leading to better translation quality.
- Hybrid NMT+RBMT can be effective: While the RBMT system itself showed lower quality scores, its integration into a multi-source NMT architecture (specifically with RNNs) provided a statistically significant improvement, demonstrating that even imperfect RBMT can contribute valuable information.
- Transfer learning from related languages is key: Leveraging data from English-Russian and Russian-Kazakh pairs was crucial for building competitive models in the low-resource English-Kazakh direction.
Key Results
- The Universitat d'Alacant's constrained submission achieved 2nd place in terms of chrF++ score on the WMT 2019 English-to-Kazakh news translation task.
- Their best performing ensemble (5 independently trained Transformer models, though not submitted due to time constraints) reached the 1st position in terms of chrF++.
- Morphological segmentation provided a slight improvement in BLEU and chrF++ scores.
- The hybrid system using Apertium with an RNN architecture outperformed the single-source RNN system, particularly in chrF++.
Limitations and Future Work
- RBMT quality: The authors noted that the Apertium output was of "very low quality," which limited the effectiveness of the multi-source Transformer architecture. They plan to explore other multi-source architectures better suited to the RBMT input.
- Ensemble size: Their submitted ensembles consisted of only 2 independent training runs. A larger ensemble of 5 Transformer models, trained independently, significantly outperformed the submitted systems, suggesting that more diverse training runs could yield even better results.
- Future work: The team plans to continue exploring the hybridization of NMT and RBMT, investigating more multi-source Transformer architectures, and using RBMT to generate synthetic training data.
Summary updated