PICT@DravidianLangTech-ACL2022: Neural Machine Translation On Dravidian Languages

Aditya Vyawahare, Rahul Tangsali, Aditya Mandke, Onkar Litake, Dipali KadamACL 2022

TL;DR

This paper tackles the challenge of building robust neural machine translation (NMT) systems for Dravidian languages, which are often low-resource. The authors achieved top performance in a shared task by experimenting with various NMT architectures and leveraging techniques like back-translation for data augmentation.

What Problem Does This Paper Address?

Developing high-accuracy machine translation systems for low-resource languages is a significant challenge, especially for language families like Dravidian. These languages often lack the vast parallel corpora needed to train state-of-the-art NMT models effectively. This paper focuses on bridging this gap by building NMT systems for Kannada to other Dravidian languages (Tamil, Telugu, Malayalam, Tulu) and also to Sanskrit, a classical Indo-Aryan language.

Methods

The researchers explored several NMT approaches:

  • Seq2Seq Models: They implemented standard encoder-decoder architectures using LSTM, bidirectional LSTM (BiLSTM), and convolutional Seq2Seq (Conv2Seq) models.
  • Transformer Models: They trained transformer models from scratch using the Fairseq library.
  • Fine-tuning Pre-trained Models: They fine-tuned a multilingual NMT model (transformer_4x from AI4Bharat) on their specific language pairs.
  • Back-translation: To combat data scarcity, they employed back-translation. This involves using monolingual data to generate synthetic parallel data, effectively augmenting the training set. They used monolingual data from indicCorp for this purpose.

Datasets

The primary datasets used were provided by the DravidianLangTech organizers, consisting of parallel corpora for five language pairs: Kannada-Tamil, Kannada-Telugu, Kannada-Malayalam, Kannada-Tulu, and Kannada-Sanskrit. These datasets were split into train, development, and test sets. For back-translation, they utilized monolingual data from indicCorp.

Key Insights and Ideas

  • Back-translation is Crucial for Low-Resource Languages: The paper highlights how back-translation significantly helps overcome data limitations, leading to improved translation accuracy.
  • Transformer Models Show Promise: While Seq2Seq models were explored, the transformer architecture, both trained from scratch and fine-tuned, demonstrated strong performance.
  • Fine-tuning Pre-trained Multilingual Models is Effective: Leveraging existing large-scale multilingual models and fine-tuning them on specific Dravidian language pairs proved to be a successful strategy.
  • Language Pair Specific Challenges: The authors noted that for very low-resource languages like Tulu and Sanskrit, some advanced multilingual models (like transformer_4x) lacked direct support, necessitating the use of custom-trained models.

Key Results

The team achieved impressive results in the DravidianLangTech-ACL2022 shared task, securing first place in three out of five sub-tasks. Specifically, they ranked first for Kannada to Malayalam, Kannada to Telugu, and Kannada to Tamil translations. They also achieved third and fourth place for Kannada to Sanskrit and Kannada to Tulu, respectively. The paper presents BLEU scores in Table 2 and Table 4, with the highest scores for each language pair marked in bold. For instance, the fine-tuned transformer model achieved a BLEU score of 0.8123 for Kannada to Tulu.

Limitations and Future Work

The authors acknowledge that for very low-resource languages like Tulu and Sanskrit, they couldn't finetune certain advanced multilingual models, leading to "Not Recorded" (NR) scores for those specific configurations.

For future work, they plan to:

  • Train models using larger-scale GPUs.
  • Explore different tokenization methods for improved training.
  • Expand the training corpora to further enhance model performance.

Summary updated

BibTeX
@inproceedings{vyawahare2022pict,
  author = {Aditya Vyawahare and Rahul Tangsali and Aditya Mandke and Onkar Litake and Dipali Kadam},
  title = {{PICT@DravidianLangTech-ACL2022: Neural Machine Translation On Dravidian Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure