TL;DR
This paper presents the IIIT-Hyderabad's Neural Machine Translation (NMT) system for the Gujarati-English news translation task at WMT19. They demonstrate that leveraging parallel data from a related high-resource language pair (Hindi-English) through multilingual NMT significantly improves translation quality for the low-resource Gujarati-English pair, achieving a notable BLEU score increase.
What Problem Does This Paper Address?
Machine translation (MT) systems, especially Neural Machine Translation (NMT), have shown remarkable performance. However, a significant challenge remains for low-resource language pairs, where the availability of parallel training data is scarce. This scarcity hinders NMT models from learning high-quality representations, leading to suboptimal translation quality. Traditional approaches to address this include using monolingual data or transfer learning from other language pairs. This paper focuses on the latter, specifically exploring how multilingual NMT, by jointly training on multiple language pairs, can boost performance for low-resource languages like Gujarati-English.
Methods
The core of the IIIT-H system is an attention-based encoder-decoder NMT model. They employ a bi-directional RNN for the encoder and a uni-directional RNN for the decoder, utilizing a global attention mechanism. For handling open vocabularies and rare words, they incorporate Byte Pair Encoding (BPE) as a subword segmentation technique.
The key innovation lies in their multilingual NMT approach. Instead of training a separate model for Gujarati-English, they leverage the existing Hindi-English parallel corpus. They combine this with the Gujarati-English parallel data to train a single multilingual model. This approach is inspired by prior work but with a simplification: they don't use artificial language tokens as the target language is consistently English.
Additionally, for their Transformer-based experiments, they explore script conversion, converting Gujarati text to the Devanagari script (used by Hindi) to facilitate better sharing of linguistic features between languages on the encoder side.
Datasets
- Gujarati-English Training Data: A collection of parallel sentences from sources like Wiki Titles, Bible corpus, Localisation Opus, Wikipedia corpus, and a crawled corpus, totaling approximately 155K parallel sentences.
- Gujarati-English Development Data:
news-dev2019. - Gujarati-English Test Data: Provided by the WMT19 organizers.
- Hindi-English Parallel Data: The IIT-Bombay parallel data (Kunchukuttan et al., 2017) was used as the assisting language pair.
Key Insights and Ideas
- Multilingual NMT for Low-Resource Gains: The most significant insight is that jointly training a single NMT model on a high-resource language pair (Hindi-English) alongside a low-resource pair (Gujarati-English) leads to substantial improvements in translation quality for the low-resource pair. This highlights the power of cross-lingual transfer learning within NMT.
- BPE's Effectiveness: Byte Pair Encoding proves to be a crucial preprocessing step, not only reducing vocabulary size but also proving more effective than simple UNK replacement for handling rare words and enabling open-vocabulary translation.
- Script Conversion as a Facilitator: For languages with similar linguistic structures but different scripts (like Gujarati and Hindi), converting the script of one language to match the other can enhance cross-lingual transfer learning, particularly when combined with subword segmentation.
Key Results
The paper reports the following BLEU scores on the WMT19 test set:
- Baseline RNN Encoder-Decoder + Attention: 6.2 BLEU
- Multilingual RNN Encoder-Decoder (using Hindi-English data): 9.8 BLEU, showing a gain of +3.6 BLEU points.
In additional Transformer experiments:
- Transformer (Base): 15.78 BLEU
- Multilingual Transformer (with averaging): 16.49 BLEU, a gain of +0.71 BLEU points.
Limitations and Future Work
- Many-to-One vs. One-to-Many: The presented multilingual NMT model is a "many-to-one" system (multiple source languages to one target language). The authors express interest in exploring "one-to-many" multilingual NMT systems in the future.
- Data Scarcity: While multilingual NMT helps, the fundamental challenge of data scarcity for low-resource languages remains. Future work could explore more advanced techniques to further mitigate this.
- Transformer Architecture: While Transformer models show higher baseline performance, the gains from multilingual training are still significant, suggesting room for further optimization in this architecture for low-resource scenarios.
Summary updated