Sheffield Submissions for WMT18 Multimodal Translation Shared Task

Chiraag Lala, Pranava Swaroop Madhyastha, Carolina Scarton, Lucia SpeciaEMNLP 2018

TL;DR

This paper presents the University of Sheffield's submissions to the WMT18 Multimodal Machine Translation shared task, focusing on improving translation quality by incorporating image information. They explore novel multimodal re-ranking strategies and data augmentation techniques, demonstrating that their approaches outperform baseline text-only models.

What Problem Does This Paper Address?

Machine translation (MT) systems, particularly neural MT (NMT), have achieved impressive performance. However, when translating sentences that describe images, text-only NMT models can struggle with ambiguity. For instance, translating "A sportsperson is playing football" into French requires knowing the gender of the sportsperson, which is often inferable from the image but not explicitly stated in the text. This paper tackles the challenge of leveraging visual information to disambiguate such cases and improve translation accuracy, especially in multimodal settings.

Methods

The core of the Sheffield team's approach lies in two main areas:

  • Multimodal Re-ranking: They build upon standard NMT systems that generate multiple translation candidates (n-best lists). These candidates are then re-ranked using novel multimodal cross-lingual word sense disambiguation (WSD) models. The idea is that these WSD models, informed by image features, can better predict which translation candidate correctly resolves ambiguities present in the source sentence.
  • Data Augmentation: For Task 1b, they explore augmenting the training data. This involves translating source sentences from other languages (French, German) into English, creating new parallel data. This augmented data is then used to train an NMT system, and a classification-based approach is employed to select the best hypothesis from the n-best list, using image and sentence embeddings as features.

They also experiment with different WSD baselines, including a Most Frequent Sense (MFS) model and a text-only Lexical Translation (LT) model, comparing them against their proposed Multimodal Lexical Translation (MLT) model.

Datasets

  • Multi30k: This dataset, provided by the task organizers, is central to their work. It contains images with corresponding English descriptions and human translations into German, French, and Czech.
  • Multimodal Lexical Translation Dataset (MLTD): This dataset, extracted from Multi30k, specifically focuses on ambiguous words and their cross-lingual translations, crucial for their WSD models.
  • WMT16 and WMT17 test sets: Used for validation purposes in Task 1.
  • WMT18 test sets: Used for evaluation in both Task 1 and Task 1b.

Key Insights and Ideas

  • Image as a Disambiguation Tool: The fundamental insight is that images can resolve textual ambiguities that text-only models miss, leading to more accurate translations.
  • N-best Lists are Rich: Even standard NMT systems produce high-quality translation hypotheses beyond the top-ranked one, making n-best list re-ranking a promising avenue.
  • Multimodal WSD Potential: Their MLT model, incorporating image features, shows slightly better performance than text-only LT models for WSD, suggesting the utility of multimodal information.
  • Data Augmentation for Task 1b: Creating synthetic parallel data through translation can be a viable strategy to improve performance, especially when dealing with complex tasks like multimodal translation.
  • Limitations of Automatic Metrics: The paper highlights that automatic metrics like BLEU, METEOR, and TER might not be sensitive enough to detect the subtle improvements gained from multimodal re-ranking, underscoring the need for human evaluation.

Key Results

  • Task 1: All submitted systems (SHEF_LT, SHEF_MLT, SHEF_MFS) outperformed the official baseline across all language pairs and metrics (METEOR, BLEU, TER). The re-ranking approaches generally showed improvements, though the difference between LT and MLT was sometimes marginal.
  • Task 1b: The submitted models also outperformed the baseline. The consensus-based selection model (SHEF_CON) achieved the best performance.
  • Re-ranking Impact: Re-ranking affected a relatively small percentage of instances and words, suggesting that while beneficial, its impact might be limited by the nature of the dataset or the ambiguity present.

Limitations and Future Work

  • Dataset Noise: The EN-CS MLTD dataset was noted as being noisy, which impacted the reliability of the trained cross-lingual WSD models for that language pair.
  • Metric Sensitivity: The authors acknowledge that automatic metrics may not fully capture the benefits of their multimodal approaches, suggesting reliance on direct human assessment for final evaluation.
  • Limited Ambiguity: The corpora used might not contain a sufficient number of ambiguous words for the multimodal WSD models to demonstrate their full potential.
  • Future Work: The paper implicitly suggests further investigation into more sensitive evaluation metrics and exploring datasets with richer ambiguity to better showcase the advantages of multimodal translation. They also mention the potential of data augmentation and classification-based selection for future work.

Summary updated

BibTeX
@inproceedings{lala2018sheffield,
  author = {Chiraag Lala and Pranava Swaroop Madhyastha and Carolina Scarton and Lucia Specia},
  title = {{Sheffield Submissions for WMT18 Multimodal Translation Shared Task}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure