TL;DR
This paper presents an efficient multilingual Neural Machine Translation (NMT) system for Indian languages, achieving significant improvements over baselines by leveraging language relatedness and strategic data selection. The system ranked second in the WAT 2021 MultiIndicMT shared task, demonstrating the effectiveness of their approach for low-resource language translation.
What Problem Does This Paper Address?
Machine translation (MT) systems, particularly Neural Machine Translation (NMT), have revolutionized language processing. However, their performance heavily relies on vast amounts of parallel training data. Many Indian languages are considered "low-resource," meaning they lack sufficient parallel corpora, which severely hampers translation quality. Traditional approaches require training separate models for each language pair, leading to complex and computationally expensive systems. This paper tackles the challenge of building high-quality, efficient multilingual NMT systems for Indian languages, aiming to overcome data scarcity and computational constraints.
Methods
The authors employ several key techniques to build their efficient multilingual NMT system:
- Unified Transliteration: To exploit the shared ancestry of Indian languages, they transliterate all Indian languages into a common script (Devanagari, the script for Hindi). This creates a unified surface form, simplifying cross-script correspondences.
- Subword Segmentation (BPE): Recognizing that Indian languages share cognates but differ at the non-root level, they use Byte Pair Encoding (BPE) to segment words into subwords. This increases vocabulary overlap at the subword level, which is crucial for related languages and helps mitigate data sparsity. BPE is applied after transliteration to ensure consistency.
- Data Selection Strategy: Instead of combining all available parallel data, which is computationally prohibitive, they adopt a data selection strategy. They incrementally add domains in decreasing order of their vocabulary overlap with a core "PMI" domain, stopping when a dip in BLEU score is observed. This ensures that only relevant data contributes to learning.
- Back Translation: This is a standard data augmentation technique where monolingual data from the target side is translated back into the source language using existing models. This synthetic data is then combined with the actual parallel data to improve language modeling and domain adaptation.
- Multilingual NMT and Fine-tuning: They utilize a "language flag" based approach (Johnson et al., 2017) where a single model handles multiple translation directions. This approach shares attention mechanisms and encoder-decoder networks, simplifying the system. They also fine-tune their multilingual system on the PMI domain, leveraging transfer learning between parent and child models.
Datasets
The experiments primarily use datasets from the WAT 2021 shared task, including:
- PMI (Haddow and Kirefu, 2020): A core dataset used for vocabulary overlap calculations and fine-tuning.
- CVIT (Siripragada et al., 2020)
- IIT-B (Kunchukuttan et al., 2017)
They also utilize monolingual data of the PMI dataset for back translation. The task itself covers 10 Indian languages (Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, and Telugu) and English, with 20 translation directions (English-Indic and Indic-English).
Key Insights and Ideas
- Leveraging Language Relatedness is Key: The core idea that Indian languages, despite their diversity, share significant lexical and structural similarities is central. Exploiting this relatedness through unified transliteration and subword segmentation is a powerful strategy for low-resource NMT.
- Strategic Data Selection Matters: Simply throwing all available data at a multilingual model isn't optimal. The authors' incremental data selection based on vocabulary overlap effectively prunes less useful data, saving computational resources and potentially improving performance.
- Multilingual Models Outperform Baselines: The results clearly show that multilingual models, even without advanced techniques, significantly outperform bilingual baselines for Indian languages. This highlights the benefit of parameter sharing.
- Combining Techniques Yields Best Results: The final system, which combines unified transliteration, subword segmentation, data selection, back translation, and fine-tuning, achieves the best performance, underscoring the synergistic effect of these methods.
Key Results
The paper reports significant improvements in BLEU scores:
- The final multilingual system outperforms baselines by an average of 11.3 BLEU points for English-Indic (en-xx) directions and 19.6 BLEU points for Indic-English (xx-en) directions.
- The multilingual system of PMI alone significantly outperforms the bilingual baseline of PMI.
- The authors' submission achieved the second rank in the WAT 2021 MultiIndicMT shared task.
Limitations and Future Work
- Computational Complexity: While the authors focus on efficiency, training large multilingual models is still computationally intensive. The data selection strategy helps, but further optimizations might be needed.
- Data Quality: The performance is still dependent on the quality and quantity of available parallel and monolingual data.
- Future Work: The paper doesn't explicitly detail future work, but the success of their approach suggests further exploration of language-specific data selection strategies, more advanced transfer learning techniques, and potentially incorporating linguistic features beyond simple transliteration and subword units. The authors also mention their submission was evaluated with AMFM scores, indicating a broader evaluation framework could be explored.
Summary updated