Rakuten’s Participation in WAT 2021: Examining the Effectiveness of Pre-trained Models for Multilingual and Multimodal Machine Translation

Raymond Hendy Susanto, Dongzhe Wang, Sunil Yadav, Mausam Jain, Ohnmar HtunACL 2021

TL;DR

This paper details Rakuten's participation in the WAT 2021 workshop, focusing on how pre-trained multilingual models like mBART can significantly boost performance across various machine translation tasks, including low-resource, multimodal, and mixed-domain scenarios. The research highlights the effectiveness of adapting these powerful pre-trained models for specific translation challenges, demonstrating their versatility and impact.

What Problem Does This Paper Address?

Machine translation (MT) systems have seen remarkable progress, largely driven by advances in neural networks and large-scale pre-training. However, applying these advancements to diverse real-world scenarios, especially those with limited parallel data (low-resource languages), mixed domains (e.g., technical documentation and news), or involving multiple modalities (like text and images), remains a significant challenge. This paper investigates how pre-trained multilingual models, specifically mBART, can be effectively fine-tuned and adapted to tackle these complex MT tasks, aiming to bridge the gap between general-purpose models and specialized translation needs.

Methods

The core of this work revolves around leveraging and adapting the mBART (Multilingual BART) pre-trained model. The authors explored several key strategies:

  • Multilingual Fine-tuning: Instead of training separate models for each language pair, they fine-tuned a single mBART model to handle multiple language directions simultaneously. This approach is particularly beneficial for low-resource languages where parallel data is scarce.
  • Domain Adaptation: For tasks involving different domains (like IT documentation and Wikinews), they used domain indicator tokens prepended to the input to guide the model.
  • Multimodal Integration: For Japanese-English multimodal translation, they compared a unimodal text-only approach (using mBART) with a multimodal approach that incorporated visual representations (Universal Visual Representation - UVR).
  • Data Augmentation: For Myanmar-English translation, they experimented with various data cleaning and tokenization strategies to improve performance on this low-resource language pair.
  • Ensemble Decoding: For several tasks, they combined the predictions of multiple independently trained models to achieve better overall performance.

Datasets

The paper utilized several datasets for training and evaluation across different tasks:

  • NICT-SAP Task:
    • Asian Language Treebank (ALT) for the Wikinews domain.
  • OPUS (GNOME, KDE4, Ubuntu) for the IT domain.
  • SAP software documentation and ALT corpus for development and evaluation.
  • Japanese↔English Multimodal Task:
    • Flickr30kEnt-JP dataset, which is an extension of Flickr30k Entities with manual Japanese translations.
  • Multilingual Indic Task:
    • Parallel corpora provided by the organizer for 10 Indic languages (Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu) paired with English.
  • PMI dataset for adaptation.
  • Myanmar-English Translation Task:
    • ALT corpus and UCSY corpus.

Key Insights and Ideas

  • Pre-training is King (Especially for Low-Resource): The paper strongly reinforces the idea that pre-trained models like mBART provide a massive head start, leading to significant performance gains, particularly for low-resource language pairs where training from scratch is often insufficient.
  • One Model to Rule Them All (Almost): A single, fine-tuned multilingual model can effectively handle multiple translation directions, offering efficiency and improved performance for languages with limited data.
  • Visuals Can Help, But Text Models Are Powerful: In multimodal settings, while incorporating visual information (UVR) showed promise, a strong text-only model fine-tuned from mBART could still achieve competitive or even superior results in certain configurations, highlighting the robustness of large pre-trained language models.
  • Data Quality Matters Immensely: For tasks like Myanmar-English translation, meticulous data preprocessing, including cleaning and appropriate tokenization, proved to be more impactful than complex model architectures or extensive data augmentation alone.

Key Results

  • NICT-SAP Task: The team achieved first place in AMFM scores for multiple translation directions on the NICT-SAP task, demonstrating the effectiveness of their mBART-based multilingual fine-tuning approach. They reported significant BLEU improvements, with one instance showing a 25.23 point gain for id→en on the ALT domain compared to a non-pre-trained baseline.
  • Japanese↔English Multimodal Task: In the constrained setting, their multimodal NMT with UVR achieved first place in BLEU and human adequacy scores. In the unconstrained setting, their text-only mBART25 finetuned model also achieved first place in BLEU and AMFM scores for the Japanese English task.
  • Multilingual Indic Task: Their mBART50-based models showed average improvements of 7.4 BLEU points over the organizer's baseline for en→in directions and 15.76 BLEU points for in→en directions. Their systems ranked second in human evaluations for several directions.
  • Myanmar-English Translation Task: Through data augmentation and preprocessing, they achieved competitive results, securing 6th place on the en-my leaderboard and 5th on my-en.

Limitations and Future Work

  • Model Size and Computational Cost: While mBART offers significant benefits, its large size and computational requirements for fine-tuning and inference can be a limitation.
  • Language Coverage: mBART50, while extensive, did not include all languages (e.g., Malay), requiring custom extensions.
  • Multimodal Integration Nuances: The paper suggests further exploration into more sophisticated ways to fuse visual and textual information for multimodal tasks.
  • Data Imbalance: While temperature-based sampling was used, addressing extreme data imbalance in multilingual settings remains an ongoing challenge.
  • Future Work: The authors implicitly suggest continued exploration of pre-trained models, advanced data augmentation, and more efficient multimodal integration techniques. The success in various tasks also points towards the potential for these models in other low-resource and specialized translation scenarios.

Summary updated

BibTeX
@inproceedings{susanto2021rakutens,
  author = {Raymond Hendy Susanto and Dongzhe Wang and Sunil Yadav and Mausam Jain and Ohnmar Htun},
  title = {{Rakuten’s Participation in WAT 2021: Examining the Effectiveness of Pre-trained Models for Multilingual and Multimodal Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure