Introduction
Multimodal Machine Translation (MMT) aims to improve translation quality by incorporating visual information from images associated with the source text. Despite its potential, MMT models often suffer from "visual laziness," where the model relies almost exclusively on textual context, ignoring the image. This paper introduces a "translate-and-refine" approach using a two-stage deliberation network to ensure the model is truly visually aware, particularly in cases of lexical ambiguity, gender neutrality, and noisy input.
Addressing Linguistic Ambiguity
Traditional text-only models struggle with specific linguistic challenges that images are uniquely suited to resolve. For instance, translating the English word "player" into a gendered language like German requires knowing the gender of the subject. Without visual cues, models often default to the most frequent training label (usually masculine). Similarly, words like "football" or "shirt" may be ambiguous or incorrectly transcribed in the source text.
Methodology: The Deliberation Network
The core of the proposed approach is a two-stage deliberation process. Instead of a single-pass translation, the model first generates a draft and then refines it.
- First Pass: A standard Transformer encoder-decoder produces a draft translation based solely on the source text.
- Second Pass: A "deliberation decoder" takes the source text, the first-pass draft, and the image as inputs. This allows the model to "proofread" the draft with the benefit of global textual context and visual grounding.
The researchers explored two types of visual representations: spatial features from CNN filter maps and object-based features derived from an off-the-shelf object detector. The latter provides a semantic bridge, allowing the model to link specific words to detected entities in the scene.
Experimental Results
The model was evaluated on the Multi30K dataset (English to German, French, and Czech). To test robustness, the authors created "degraded" datasets by masking ambiguous words (AMB), random words (RND), and gender-specific terms (PERS).
The results demonstrated that the deliberation-based multimodal models achieved state-of-the-art performance. Specifically, the inclusion of object-based features (del+obj) significantly improved METEOR scores on degraded tasks. For example, on the German test set with masked ambiguous words, the model improved the score from 48.4 to 49.8. Human evaluations also favored the deliberation models, particularly for French translations.
Conclusion and Future Work
The deliberation network effectively mitigates the "laziness" problem in MMT by providing a dedicated stage for visual integration. While the model shows improved gender-marking and robustness to noise, it remains susceptible to errors in object detection and increased computational costs. Future research will likely focus on tighter integration between visual and textual entities to further reduce "hallucinated" translations.
Summary updated