Adversarial Evaluation of Multimodal Machine Translation

Desmond ElliottEMNLP 2018 · OpenReview

Multimodal Machine Translation (MMT) aims to enhance translation quality by integrating visual context—typically an image—alongside a source sentence. The core hypothesis is that visual information can resolve linguistic ambiguities, such as determining whether the word "crane" refers to a bird or a piece of construction equipment. However, despite the intuitive appeal, MMT research has often yielded inconsistent results, with many multimodal models failing to significantly outperform text-only baselines. This raises a critical question: are these models truly leveraging visual information, or is the image input merely acting as a complex form of regularization?

In the paper "Adversarial Evaluation of Multimodal Machine Translation" (EMNLP 2018), Desmond Elliott introduces a simple yet powerful adversarial framework to measure the "image awareness" of MMT systems. By systematically perturbing the visual input, the study reveals that many state-of-the-art models are surprisingly indifferent to the images they are supposedly processing.

The Adversarial Evaluation Framework

To move beyond the "black box" nature of MMT systems, Elliott proposes an adversarial setup that compares model performance under two distinct conditions:

  1. Congruent Setting: The model receives the correct source sentence ($x$) and its corresponding image ($v$).
  2. Incongruent Setting: The model receives the same source sentence ($x$) but is paired with a randomly selected image ($\bar{v}$) from the dataset.
Figure 6: Examples of the adversarial evaluation. In (a), the congruent image helps the model achieve a higher Meteor score. In (b), the model performs slightly better with an incongruent image, suggesting a lack of visual grounding.

The difference in performance between these two settings defines the model's $\Delta$-Awareness. This is measured in two ways:

  • Model-internal awareness ($\Delta_I$): The change in the log-probability the model assigns to the ground-truth translation. An aware model should find the correct translation more probable when the correct image is present.
  • Model-external awareness ($\Delta_E$): The change in standard translation metrics, specifically the Meteor score. A significant drop in Meteor score when presented with a random "distractor" image indicates that the model was relying on the visual context for its output.

Experimental Setup and Results

The study utilized the Multi30K dataset, the standard benchmark for MMT, consisting of English images and their German translations. Three representative MMT architectures were evaluated:

  • trgmul: Modulates word embeddings with visual features.
  • decinit: Uses the image to initialize the hidden state of the decoder.
  • hierattn: Employs a hierarchical attention mechanism to focus on specific spatial regions of the image.

Key Findings

The results, summarized in the table below, highlight a stark contrast in how different architectures handle visual data.

System Congruent (Meteor) Incongruent (Meteor) $\Delta_E$-Awareness
trgmul 57.3 57.3 -0.001 (Failed)
decinit 57.0 56.8 0.003 (Weak)
hierattn 55.0 53.3 0.019 (Strongest)

As shown in the distribution of Meteor score differences, hierattn was the only system to demonstrate statistically significant sensitivity to the visual input. In contrast, trgmul showed zero sensitivity, suggesting that the visual signal was effectively ignored during the decoding process.

Figure 4: Distribution of the difference in Meteor scores between congruent and incongruent settings across the three models. The wider spread and higher mean for `hierattn` indicate greater sensitivity to visual context.

The "Illusion" of Multimodality

The most striking insight from this research is the "illusion" of multimodality. Simply providing an image input does not guarantee its use. Models that rely on global, average-pooled image features (like trgmul and decinit) are highly prone to ignoring the visual modality in favor of the stronger linguistic signal. Spatial attention, as seen in hierattn, appears to be a prerequisite for meaningful visual grounding.

Furthermore, the paper identifies a significant data bottleneck. Analysis suggests that only approximately 13% of the Multi30K dataset actually requires the image for a human to produce an accurate translation. If the text is sufficient 87% of the time, the model naturally learns to bypass the more complex visual features. This suggests that current MMT benchmarks may not be challenging enough to force models to become truly multimodal.

Conclusion and Future Directions

Elliott’s adversarial evaluation provides a necessary reality check for the MMT community. It demonstrates that high translation scores do not necessarily imply visual awareness. To advance the field, the author suggests several paths forward:

  1. Visually Grounded Datasets: Creating datasets where the image is indispensable for translation (e.g., through post-editing to ensure visual dependency).
  2. Adversarial Training: Incorporating the incongruent setting into the training objective to force models to distinguish between relevant and irrelevant visual context.
  3. Broader Application: Extending this awareness check to other multimodal tasks, such as Visual Question Answering (VQA), to ensure models are not simply exploiting linguistic biases.

Summary updated

BibTeX
@inproceedings{elliott2018adversarial,
  author = {Desmond Elliott},
  title = {{Adversarial Evaluation of Multimodal Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=S1bDaZf_ZB},
}
Figure