Grounded Word Sense Translation

Chiraag Lala, Pranava Madhyastha, Lucia SpeciaNAACL 2019

TL;DR

This paper tackles the challenge of "grounded word sense translation," where the goal is to correctly translate an ambiguous word in a sentence, using both the sentence's text and an accompanying image. They propose treating this as a sequence labeling task and explore various multimodal models, finding that object-based grounding significantly improves translation accuracy, especially when focusing on explicit object information.

What Problem Does This Paper Address?

Machine translation (MT) systems have made great strides, but they still struggle with ambiguity, particularly when a word has multiple meanings. Traditional MT relies solely on textual context, which can be insufficient for disambiguating words. This paper focuses on "grounded word sense translation" (MLT), a more nuanced task that leverages visual information (an image) alongside textual context to resolve word ambiguity. The core problem is to determine how much and in what way images can improve the translation of these ambiguous words, moving beyond sentence-level improvements to focus on lexical accuracy.

Methods

The authors frame multimodal lexical translation (MLT) as a sequence labeling task, similar to part-of-speech tagging. They adapt existing neural sequence architectures, specifically unidirectional (ULSTM) and bidirectional (BLSTM) Long Short-Term Memory networks, to incorporate visual information.

Their multimodal approaches include:

  • Multimodal Tagger: Initializing the LSTM hidden states with image features extracted from a pre-trained ResNet-50 model.
  • Object-based Grounding: This is a novel approach where they leverage explicit object information from images. They experiment with two strategies:
  • Initializing: Similar to the multimodal tagger, but using a binary vector representing object presence.
  • Prepending: Prepending the name of an object category (e.g., "person") to the source sentence. This draws inspiration from recent work in neural machine translation.

They also investigate different data settings for training, such as using only sentences with ambiguous words versus all sentences, and whether to label all words or only ambiguous ones.

Datasets

The primary dataset used is the Multimodal Lexical Translation (MLT) dataset, which was extracted from the Multi30k dataset. This dataset consists of images, English descriptions, and for specific ambiguous words in the descriptions, their corresponding translations in the target language (German or French). The paper also mentions using the 2018 test set of the Multimodal Machine Translation (MMT) shared task for human annotation experiments.

Key Insights and Ideas

  • Images Help, Especially for Weaker Models: The study reveals that visual information provides a significant boost, particularly for simpler unidirectional ULSTM models. This suggests images can compensate for less sophisticated textual context processing.
  • Object-Based Grounding is Key: Explicitly grounding the translation of ambiguous words to specific objects in the image (rather than just general image features) yields the most substantial improvements. This highlights the importance of fine-grained visual information.
  • Prepending Object Information is Superior: Among object-based grounding methods, prepending the object category name to the input sentence proved to be the most effective strategy.
  • Data Settings Matter: Training on "ambiguous sentences" and labeling "all words" (both ambiguous and unambiguous) generally led to better performance, suggesting that focusing the model on relevant data and providing consistent labeling is beneficial.
  • Skewed Distributions are a Challenge: The dataset exhibits highly skewed distributions of translation candidates for ambiguous words, making it difficult to outperform the most frequent translation. This inherent challenge underscores the value of any improvement.

Key Results

  • Human annotators found images useful for disambiguation in 7.8% (EnDe) to 8.6% (EnFr) of cases, indicating a tangible scope for image-based improvements.
  • Object-based grounding models, particularly BLSTM+object-prepend, outperformed general multimodal models (using ResNet features) by a significant margin. For instance, gains over text-only baselines ranged from 1.41% to 2.52% for EnDe and 1.62% to 4.00% for EnFr.
  • The best multimodal model, BLSTM+object-prepend, trained in the "ambiguous sentences" and "all words" setting, improved accuracy by 1.41% for EnDe and 2.24% for EnFr over the best text-only baseline.
  • ULSTM models showed a greater benefit from image features compared to BLSTM models, with gains of up to 2.75% (EnDe) and 2.47% (EnFr) in certain settings.

Limitations and Future Work

  • Dataset Skewness: The paper acknowledges the significant challenge posed by the skewed distribution of translation candidates in the dataset, which makes it difficult to demonstrate improvements over the most frequent translation.
  • Sample Size for Human Study: While informative, the sample size for the human experiment on image utility was relatively small.
  • Future Work: The authors suggest that further exploration of object-based grounding, potentially with more sophisticated object detection and representation methods, holds promise. They also hint at the potential for exploring other forms of structured visual information.

Summary updated

BibTeX
@inproceedings{lala2019grounded,
  author = {Chiraag Lala and Pranava Madhyastha and Lucia Specia},
  title = {{Grounded Word Sense Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure