NLPHut’s Participation at WAT2021

Shantipriya Parida, Subhadarshi Panda, Ketan Kotwal, Amulya Ratna Dash, Satya Ranjan Dash, Yashvardhan Sharma, Petr Motlicek, Ondřej BojarACL 2021

TL;DR

This paper details NLPHut's participation in the WAT2021 workshop, focusing on multimodal and multilingual translation tasks. They achieved top results in English-Malayalam multimodal translation and second-best in English-Hindi multimodal translation by leveraging Transformer models and a novel region-specific image captioning approach.

What Problem Does This Paper Address?

The paper addresses the ongoing challenges in machine translation (MT), particularly for low-resource languages and the integration of multiple modalities (like images) to improve translation quality. While MT has made significant strides, effectively translating between languages with limited data and incorporating visual context remains an active research area. The WAT2021 workshop, specifically its multimodal and multilingual tracks, provides a platform to tackle these issues by introducing new datasets and tasks, such as multimodal translation involving image captioning and translation for various Indic languages.

Methods

NLPHut employed the state-of-the-art Transformer model for their translation tasks, utilizing language tags to guide the translation process in multilingual settings. For the multimodal tasks, they combined text-only translation with image captioning. A key innovation is their region-specific image captioning approach. Instead of captioning the entire image, they focus on specific regions defined by bounding boxes. This method fuses features from both the region of interest and the entire image, using a weighted combination, to provide richer context for generating more accurate and relevant captions. The image encoder uses features from a pre-trained CNN, and an LSTM-based decoder generates the captions. For multilingual translation, they explored various setups including many-to-one, one-to-many, and many-to-many configurations, with and without language tags, to optimize performance across multiple Indic languages.

Datasets

The following datasets were used:

  • English→Hindi Multimodal Translation:
    • HindiVisualGenome 1.1 (HVG): A dataset of English and Hindi short captions for image regions.
  • IITB Corpus: A large parallel English-Hindi corpus.
  • English→Malayalam Multimodal Translation:
    • MalayalamVisualGenome 1.0 (MVG): An extension of HVG for English and Malayalam.
  • Indic Multilingual Translation:
    • A training corpus comprising 11 million sentence pairs from various sources.
  • Filtered data from the PMIndia dataset for evaluation.

Key Insights and Ideas

  • Region-specific captioning is crucial for context: Captioning only a small region of an image without considering the broader context can lead to less meaningful descriptions. Fusing region-specific features with global image features significantly improves caption quality.
  • Language tags enhance multilingual translation: Explicitly informing the model about source and target languages (using language tags) generally leads to better translation performance in multilingual settings, especially for Indic languages.
  • Transformer architecture is robust: The Transformer model, a standard in NLP, proved effective across all tasks, demonstrating its versatility for both translation and multimodal applications.
  • Transfer learning and multi-phase training are beneficial: For multilingual tasks, a two-phase training approach (general multilingual training followed by language-pair specific fine-tuning) showed promise.

Key Results

NLPHut achieved significant success at WAT2021:

  • English→Malayalam Multimodal Translation: Ranked best in both text-only translation and Malayalam image captioning.
  • English→Hindi Multimodal Translation: Ranked second-best in text-only translation and Hindi image captioning.
  • Indic Multilingual Translation: Achieved competitive results, with their system often outperforming the best competitor in specific language pairs, particularly when using target language tags. For example, they achieved the highest development BLEU scores for 8 out of 10 Indic languages when translating to English using source and target tags.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Model Size: The Transformer models used could be scaled up to match larger architectures (like Transformer-Big) for potentially better performance.
  • Data Utilization: Exploring the use of all available training data for the initial phase of multilingual training could yield better results.
  • Region-Specific Captioning: Future work includes fine-tuning the object detection model used within their region-specific image captioning system.
  • Indic Multilingual Translation: They plan to further explore this task by incorporating more data and additional resources for training.

Summary updated

BibTeX
@inproceedings{parida2021nlphuts,
  author = {Shantipriya Parida and Subhadarshi Panda and Ketan Kotwal and Amulya Ratna Dash and Satya Ranjan Dash and Yashvardhan Sharma and Petr Motlicek and Ondřej Bojar},
  title = {{NLPHut’s Participation at WAT2021}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure