IITP at WAT 2021: System description for English-Hindi Multimodal Translation Task

Baban Gain, Dibyanayan Bandyopadhyay, Asif EkbalACL 2021

TL;DR

This paper presents a multimodal neural machine translation (NMT) system for English-Hindi translation, leveraging image information to disambiguate text and improve translation quality, especially for low-resource languages. The system achieved competitive results in the WAT 2021 challenge, demonstrating the value of multimodal approaches.

What Problem Does This Paper Address?

Traditional Neural Machine Translation (NMT) systems primarily rely on textual data. While effective, they often struggle with low-resource language pairs and can exhibit ambiguity, particularly when a word has multiple meanings (e.g., "cricket" as a sport or an insect). This ambiguity arises because NMT models lack "language grounding" – the ability to connect language to real-world concepts, which humans achieve through interaction with the world. Multimodal NMT aims to bridge this gap by incorporating information from other modalities, such as images, to provide context and disambiguate meaning, thereby improving translation accuracy. This paper specifically addresses the challenge of English-Hindi translation, a low-resource pair where visual context can be particularly beneficial.

Methods

The authors employ a standard NMT architecture using a Bidirectional RNN encoder and a doubly attentive RNN decoder, implemented within the OpenNMT-py framework. Their approach focuses on incorporating visual features alongside text. For image processing, they use pre-trained VGG19-bn to extract image representations from cropped portions of images, aiming to discard irrelevant visual information.

They explore two training strategies:

  1. With Pre-training: A model is first pre-trained on a large English-Hindi parallel corpus (HindEnCorp) without visual features. This pre-trained model is then fine-tuned on the multimodal dataset (Hindi Visual Genome 1.1) which includes images.
  2. Without Pre-training: The model is directly fine-tuned on the multimodal dataset, using both textual and visual features from the start.

During inference, they combine hypotheses from both models and select the one with the best log-likelihood after filtering out <unk> tokens.

Datasets

  • Hindi Visual Genome 1.1: This is the primary multimodal dataset used for training and evaluation. It contains 28,929 parallel English-Hindi sentence pairs, each associated with an image. The images often depict specific portions of a scene, and the task is to translate the description of that portion.
  • HindEnCorp: A large English-Hindi parallel corpus of 273,000 sentence pairs, used for pre-training the model. This dataset does not contain images.

Key Insights and Ideas

  • Visual Grounding for Ambiguity Resolution: The core idea is that visual information can resolve semantic ambiguities in text, leading to more accurate translations. For instance, an image can clarify whether "cricket" refers to the game or the insect.
  • Low-Resource Language Pair Advantage: Multimodal NMT is particularly promising for low-resource language pairs like English-Hindi, where textual data is scarce, but visual representations are often universal.
  • Pre-training Boosts Performance: Pre-training on a large text-only corpus before fine-tuning on the multimodal data appears to be a beneficial strategy for improving translation quality.
  • Challenge Set Design: The challenge set was specifically curated to include ambiguous English words where images are crucial for disambiguation, making it a more difficult translation task compared to a randomly selected evaluation set.

Key Results

The system achieved competitive results in the WAT 2021 English-Hindi multimodal translation task:

  • Evaluation Set: 42.47 BLEU points, securing second place.
  • Challenge Set: 37.50 BLEU points.

The paper highlights that their system's translation for "A man climbing into a train" was more nuanced than Google NMT's, demonstrating better contextual understanding.

Limitations and Future Work

  • Challenge Set Difficulty: The authors acknowledge that the challenge set's specific design for ambiguity resolution made it a harder task, potentially explaining the lower BLEU score compared to the evaluation set.
  • Utterance Length Discrepancy: A difference in utterance length between training and testing data for the challenge set is also cited as a potential reason for lower performance.
  • Future Work: The authors plan to extend their work by training with more monolingual data and exploring more advanced methods for incorporating multimodal features. They also aim to develop models that can better handle the nuances of low-resource language pairs.

Summary updated

BibTeX
@inproceedings{gain2021iitp,
  author = {Baban Gain and Dibyanayan Bandyopadhyay and Asif Ekbal},
  title = {{IITP at WAT 2021: System description for English-Hindi Multimodal Translation Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure