Language Relatedness and Lexical Closeness can help Improve Multilingual NMT: IITBombay@MultiIndicNMT WAT2021

Jyotsana Khatri, Nikhil Saini, Pushpak BhattacharyyaACL 2021

TL;DR

This paper presents a multilingual Neural Machine Translation (NMT) system for 10 Indic languages and English, focusing on improving translation quality for low-resource languages. The authors demonstrate that leveraging language family relatedness and using transliteration to a common script significantly boosts performance by reducing lexical gaps and enabling better parameter sharing.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) has revolutionized translation, but its effectiveness is heavily dependent on the availability of large parallel datasets. Many language pairs, especially those involving less common languages (low-resource languages), lack sufficient parallel data, hindering NMT performance. Multilingual NMT offers a promising solution by training a single model for multiple languages, allowing knowledge transfer from high-resource pairs to low-resource ones. However, effectively leveraging this multilingual setting, particularly for languages with diverse scripts and varying degrees of relatedness like the Indic languages, remains a challenge. This paper specifically addresses how to improve multilingual NMT for Indic languages by exploiting linguistic similarities and mitigating script differences.

Methods

The core of the proposed system is a multilingual NMT model with a shared encoder-decoder Transformer architecture. To handle the diversity of Indic languages, the authors employ two key strategies:

  • Transliteration to a Common Script: All Indic language data is transliterated into a common script, Devanagari. This is an arbitrary choice but serves to reduce the "lexical gap" between languages that use different scripts, thereby creating a more unified vocabulary for the model.
  • Language Family Grouping: The authors explore the benefit of grouping languages based on their linguistic families (Indo-Aryan and Dravidian). This allows for training specialized multilingual models that leverage the inherent relatedness within these families, hypothesizing that this leads to better parameter sharing and reduced confusion compared to a single, undifferentiated multilingual model.

Additionally, the system incorporates language embeddings added to each token in both the encoder and decoder. These embeddings are learned during training and help the model distinguish between languages, further enhancing its ability to handle multilingual input and output.

Datasets

The primary dataset used is the parallel data provided for the MultiIndicMT: An Indic Language Multilingual Task at WAT 2021. This task involves 10 Indic languages (Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, and Telugu) and English. The paper also mentions the use of monolingual data for potential future work.

Key Insights and Ideas

  • Language Family Matters: Grouping languages by their family (Indo-Aryan and Dravidian) significantly improves NMT performance compared to a single, undifferentiated multilingual model. This suggests that exploiting inherent linguistic similarities within families is more effective than treating all languages equally.
  • Transliteration Bridges the Gap: Converting Indic languages to a common script (Devanagari) effectively reduces the lexical gap, leading to a larger shared vocabulary and improved translation quality. This is crucial for languages that, despite being related, use different writing systems.
  • Multilingual NMT Outperforms Bilingual: The multilingual models, especially when leveraging language family grouping, consistently outperform simpler bilingual models, demonstrating the power of knowledge transfer in low-resource scenarios.
  • Shared Vocabulary Benefits: The use of shared subword vocabularies (learned via Byte Pair Encoding) across languages, combined with transliteration, further enhances the model's ability to generalize and learn from shared linguistic structures.

Key Results

The paper reports significant improvements in BLEU scores. On average, grouping languages by family (IA-En & DR-En) led to a 14% increase in BLEU scores compared to bilingual models. Combining all languages into a single multilingual model (All-En) yielded a 7% improvement on average. For specific language pairs, the IA-En and DR-En models showed better performance than the All-En model, highlighting the advantage of family-specific grouping. The paper also presents Adequacy-Fluency Metrics (AM-FM) which further support the effectiveness of the multilingual approaches.

Limitations and Future Work

The authors acknowledge that their models were trained for a fixed number of epochs, which does not guarantee convergence. Due to resource constraints, they were unable to train models until convergence.

For future work, the authors plan to utilize monolingual data to further improve multilingual NMT. This is a common and effective strategy for low-resource scenarios, as monolingual data is often more abundant than parallel data. They also suggest exploring other grouping strategies beyond language families.

Summary updated

BibTeX
@inproceedings{khatri2021language,
  author = {Jyotsana Khatri and Nikhil Saini and Pushpak Bhattacharyya},
  title = {{Language Relatedness and Lexical Closeness can help Improve Multilingual NMT: IITBombay@MultiIndicNMT WAT2021}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure