NICT-5’s Submission To WAT 2021: MBART Pre-training And In-Domain Fine Tuning For Indic Languages

Raj Dabre, Abhisek ChakrabartyACL 2021

TL;DR

This paper explores the effectiveness of pre-training a multilingual model (MBART) on a large collection of monolingual data and then fine-tuning it on smaller, in-domain parallel corpora for Indic languages. The authors demonstrate that this approach significantly boosts translation quality, outperforming models trained solely on parallel data, especially in low-resource scenarios.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) systems excel when large parallel datasets are available. However, for many language pairs, particularly those involving Indic languages, such datasets are scarce. This scarcity leads to poor translation quality due to a lack of learned translation knowledge and overfitting. Existing solutions like cross-lingual transfer and multilingual training have limitations, often relying on shared languages or extensive parallel data. This paper tackles the challenge of improving NMT for low-resource Indic languages by leveraging the abundance of monolingual data through pre-training.

Methods

The core of this work lies in combining two powerful techniques:

  • MBART Pre-training: The authors utilize the MBART (Multilingual BART) model, a denoising autoencoder pre-trained on a massive amount of monolingual text across 25 languages. This pre-training phase imbues the model with a strong understanding of language structure and modeling. The pre-training process involves "corrupting" input sentences by masking random spans and training the model to reconstruct the original sentence.
  • Multilingual Fine-tuning: After pre-training, the MBART model is fine-tuned on the specific Indic language pairs for the WAT 2021 task. They explore two fine-tuning strategies:
  • Bilingual Fine-tuning: Training separate models for each language pair.
  • Multilingual Fine-tuning: Training a single model to handle multiple language pairs simultaneously.

They also establish baselines by training multilingual models from scratch without any pre-training.

Datasets

The study uses data for the "MultiIndicMT" task, which involves translation between 10 Indic languages and English.

  • Monolingual Corpora: The AI4Bharat's IndicCorp dataset was used for MBART pre-training. This dataset covers Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu, and English, plus Assamese.
  • Parallel Corpora: The official parallel corpora provided by the organizers for the WAT 2021 task were used for fine-tuning and evaluation. This includes the PMI dataset for bilingual corpora statistics.

Key Insights and Ideas

  • Monolingual Data is Gold: The paper strongly advocates for the utility of monolingual data, especially when parallel data is scarce. MBART's pre-training on this abundant resource is shown to be a significant advantage.
  • Pre-training Pays Off: Even a relatively short pre-training phase (around one epoch) followed by fine-tuning yields substantial gains over models trained from scratch.
  • Multilingualism is Key: Multilingual models consistently outperform unidirectional models, highlighting the benefits of shared representations across languages.
  • Fine-tuning Enhances Performance: Multilingual fine-tuning of the pre-trained MBART model leads to the best translation quality, demonstrating the synergy between pre-training and task-specific adaptation.
  • Indic to English vs. English to Indic: The authors observe that multilingualism benefits Indic-to-English translation more than English-to-Indic translation, suggesting potential asymmetries in language pair difficulty or data characteristics.

Key Results

The paper reports significant improvements in translation quality using their MBART pre-trained and multilingual fine-tuned models. While specific BLEU scores are presented in Table 3, the key takeaway is that their best models significantly outperform baselines that do not leverage pre-training. For instance, their MBART+Many-to-one models achieve considerably higher BLEU scores across most language pairs compared to non-pre-trained multilingual models. The official best submission scores are also shown for comparison, indicating that while their approach is competitive, further advancements are possible with more sophisticated techniques like data selection and script mapping.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Pre-training Epochs: The MBART model was trained for only about one epoch due to time constraints. They believe a larger model with more extensive pre-training could yield even better results.
  • No Hyperparameter Tuning: They did not perform extensive hyperparameter tuning for fine-tuning, suggesting that optimization in this area could further improve performance.
  • Data Selection and Script Mapping: Their approach did not involve advanced techniques like data selection or script mapping, which were used by other top-performing submissions.
  • Small Parallel Corpora: The fine-tuning was done on the PMI dataset, which has relatively small parallel corpora for some pairs.

Future work could involve training with larger parallel corpora, incorporating data selection and script mapping techniques, and exploring more extensive hyperparameter tuning and model scaling.

Summary updated

BibTeX
@inproceedings{dabre2021nict,
  author = {Raj Dabre and Abhisek Chakrabarty},
  title = {{NICT-5’s Submission To WAT 2021: MBART Pre-training And In-Domain Fine Tuning For Indic Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure