Multilingual Machine Translation Systems at WAT 2021: One-to-Many and Many-to-One Transformer based NMT

Shivam Mhaskar, Aditya Jain, Aakash Banerjee, Pushpak BhattacharyyaACL 2021

TL;DR

This paper presents two Transformer-based multilingual Neural Machine Translation (NMT) systems for the WAT 2021 MultiIndicMT task, covering English-to-Indic and Indic-to-English translation for 10 Indic languages. The authors demonstrate that their multilingual models significantly outperform bilingual baselines, especially for low-resource language pairs, highlighting the benefits of transfer learning.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) systems have revolutionized translation, but they typically require large amounts of parallel data for each language pair. This poses a significant challenge for "low-resource" languages, where such data is scarce. Multilingual NMT (MNMT) systems aim to address this by leveraging the relationships between languages to improve translation quality, even for pairs with limited or no direct parallel training data (zero-shot translation). Furthermore, a single MNMT system can drastically reduce the computational and storage overhead compared to training individual bilingual models for every language pair.

Methods

The authors implemented two distinct MNMT architectures based on the Transformer model:

  • One-to-Many: This system translates from a single source language (English) to multiple target languages (10 Indic languages). They employed a single shared encoder and a single shared decoder. A language-specific token is prepended to the input sentence to signal the desired target language. This approach allows representations learned from one language pair to benefit others.
  • Many-to-One: This system translates from multiple source languages (10 Indic languages) to a single target language (English). Similar to the one-to-many approach, they used a single shared encoder and a single shared decoder. Since the target language is constant, prepending a specific token is optional.

Both models utilize a shared vocabulary for the Indic languages and a separate vocabulary for English, depending on whether it's the source or target. Byte Pair Encoding (BPE) was used for sub-word segmentation, which is particularly beneficial for morphologically rich Indic languages.

Datasets

The systems were trained and evaluated using the datasets provided for the WAT 2021 MultiIndicMT task. No additional external data was used. The paper includes a table (Table 1) detailing the number of parallel sentences for each English-Indic language pair across various datasets. The validation and test sets consisted of 1,000 and 2,390 sentences, respectively, and were 11-way parallel.

Key Insights and Ideas

  • Transfer Learning is Powerful: The core idea is that by training on multiple related languages simultaneously, the model can learn more robust and generalizable representations. This "transfer learning" significantly boosts performance, especially for languages with less training data.
  • Low-Resource Languages Benefit Most: The paper's results strongly suggest that low-resource language pairs experience a much greater improvement in translation quality when using MNMT compared to high-resource pairs. This is a crucial finding for expanding MT capabilities to under-served languages.
  • Efficiency Gains: A single multilingual model can replace dozens of individual bilingual models, leading to substantial reductions in model size and training complexity.
  • Shared Encoder/Decoder Synergy: The use of a single shared encoder and decoder allows for implicit knowledge sharing between language pairs, where learning to translate one pair can positively influence the translation of another.

Key Results

The paper reports significant improvements in translation quality for both one-to-many and many-to-one systems compared to bilingual baselines.

  • One-to-Many: For English to Indic languages, the MNMT model achieved better BLEU and RIBES scores across all language pairs. The improvement was particularly notable for English-Kannada (up to 8.93 BLEU score increase) and English-Oriya.
  • Many-to-One: For Indic languages to English, the MNMT model also outperformed bilingual baselines. For example, Gujarati to English saw an increase from 31.92 BLEU (baseline) to 35.31 BLEU (one-to-many).

Limitations and Future Work

  • Limitations: The paper does not explicitly discuss limitations beyond the inherent challenges of low-resource translation. The performance gains, while significant, are still subject to the quality and quantity of available parallel data.
  • Future Work: The authors suggest that further exploration of multilingual architectures and training strategies could lead to even greater improvements. The paper implicitly points towards the continued development of MNMT for bridging language barriers, especially for under-represented languages.

Summary updated

BibTeX
@inproceedings{mhaskar2021multilingual,
  author = {Shivam Mhaskar and Aditya Jain and Aakash Banerjee and Pushpak Bhattacharyya},
  title = {{Multilingual Machine Translation Systems at WAT 2021: One-to-Many and Many-to-One Transformer based NMT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure