Abstract
Current multimodal machine translation (MMT) systems rely on fully supervised data (i.e sentences with their translations and accompanying images), which is costly to collect and prevents the extension of MMT to language pairs with no such data. We propose a method to bypass the need for fully supervised data to train MMT systems, using multimodal English data only. Our method (ZeroMMT) consists in adapting a strong text-only machine translation (MT) model by training it jointly on two objectives: visually conditioned masked language modelling and the Kullback-Leibler divergence between the original MT and new MMT outputs. We evaluate on standard MMT benchmarks and on CoMMuTE, a contrastive test set designed to evaluate how well models use images to disambiguate translations. ZeroMMT obtains disambiguation results close to state-of-the-art MMT models trained on fully supervised examples. To prove that ZeroMMT generalizes to languages with no fully supervised training data, we extend CoMMuTE to three new languages: Arabic, Russian and Chinese. We also show that we can control the trade-off between disambiguation capabilities and translation fidelity at inference time using classifier-free guidance and without any additional data. Our code, data and trained models are publicly accessible.
Introduction
Translating a sentence like "The crane is by the water" is impossible to do perfectly without context—is it a bird or a piece of construction equipment? Multimodal Machine Translation (MMT) solves this by using images to disambiguate text. However, current MMT systems are bottlenecked by data. They typically require "triplets": an image, a source sentence, and a human-authored translation in the target language. Collecting these is incredibly expensive; for instance, translating just 30,000 captions into German can cost roughly €23,000. Consequently, MMT has been largely restricted to high-resource languages like French and German. ZeroMMT addresses this by building effective MMT systems for any language using only monolingual multimodal data (English image-caption pairs) and existing text-only translators.
Methods
ZeroMMT transforms a pretrained text-only model (NLLB) into a multimodal one through a two-stage adaptation process. The architecture incorporates lightweight adapters and a visual projector into the frozen NLLB model. Image features are extracted using SigLIP and concatenated to text embeddings.
The training relies on two primary objectives:
- Visually Conditioned Masked Language Modeling (VMLM): The model is presented with an English caption containing masked words and the corresponding image. It must use the visual context to predict the missing words, forcing the model to learn that the visual modality contains necessary disambiguating information.
- Kullback-Leibler (KL) Divergence: To prevent the model from losing its translation capabilities, the original text-only MT model acts as a "teacher." The MMT model is trained to match the output distribution of the original model when translating captions into target languages.
Additionally, the researchers implemented Classifier-Free Guidance (CFG) at inference time. By calculating the difference between a visually-grounded translation and a text-only translation, users can adjust a guidance scale ($\gamma$) to prioritize either visual disambiguation or general translation fidelity without retraining the model.
Datasets and Evaluation
ZeroMMT was trained on the Conceptual Captions dataset, which contains approximately 2.8 million English image-caption pairs. Evaluation was conducted using the standard Multi30K benchmark and CoMMuTE, a contrastive evaluation set specifically designed to test visual disambiguation of lexical ambiguity. A significant contribution of this work is the extension of the CoMMuTE benchmark to three new languages—Arabic, Russian, and Chinese—enabling the first zero-shot multimodal evaluation for these pairs.
Key Results
ZeroMMT achieved disambiguation scores on CoMMuTE that rival state-of-the-art supervised models. For English-to-French, it reached 65.0 accuracy (with CFG), significantly outperforming the 50.0 random baseline of text-only models. Remarkably, the model performed well on Arabic, Russian, and Chinese, despite seeing zero multimodal training examples for these languages.
Crucially, ZeroMMT maintained nearly the same BLEU and COMET scores as the original NLLB model on standard non-ambiguous benchmarks. This indicates that adding visual capabilities does not degrade the underlying translator's performance, a common issue in previous MMT research.
Discussion and Future Work
The success of ZeroMMT highlights that cross-lingual visual grounding is possible: learning to associate English words with visual concepts transfers effectively to other languages within the same multilingual model. However, a small gap remains between ZeroMMT and models trained with full supervision on specific domains. Future work will focus on "ambiguity detection" to determine when a sentence actually requires visual context, thereby reducing computational overhead and minimizing the risk of visual hallucinations when the text is already clear.
Summary updated