Abstract
Current multimodal machine translation (MMT) systems rely on fully supervised data (i.e models are trained on sentences with their translations and accompanying images). However, this type of data is costly to collect, limiting the extension of MMT to other language pairs for which such data does not exist. In this work, we propose a method to bypass the need for fully supervised data to train MMT systems, using multimodal English data only. Our method, called ZeroMMT, consists in adapting a strong text-only machine translation (MT) model by training it on a mixture of two objectives: visually conditioned masked language modelling and the Kullback-Leibler divergence between the original and new MMT outputs. We evaluate on standard MMT benchmarks and the recently released CoMMuTE, a contrastive benchmark aiming to evaluate how well models use images to disambiguate English sentences. We obtain disambiguation performance close to state-of-the-art MMT models trained additionally on fully supervised examples. To prove that our method generalizes to languages with no fully supervised training data available, we extend the CoMMuTE evaluation dataset to three new languages: Arabic, Russian and Chinese. We further show that we can control the trade-off between disambiguation capabilities and translation fidelity at inference time using classifier-free guidance and without any additional data. Our code, data and trained models are publicly accessible.
Introduction
Multimodal Machine Translation (MMT) aims to integrate visual context to resolve linguistic ambiguities that text alone cannot handle. For instance, the word "crane" in English could refer to a bird or construction machinery; an accompanying image provides the necessary disambiguation signal. However, the development of MMT systems is severely constrained by the scarcity of "fully supervised" datasets—images paired with human-translated captions in multiple languages. The cost of such data collection (e.g., €23,000 for 30,000 German captions) limits MMT to a few high-resource languages. This research introduces ZeroMMT, a framework designed to build multimodal translators using only monolingual image-text pairs, bypassing the need for expensive parallel multimodal data.
Methodology
ZeroMMT "upgrades" a strong pre-trained text-only machine translation model, specifically No Language Left Behind (NLLB), into a multimodal system. The architecture keeps the original NLLB weights frozen, adding lightweight adapters and a visual projector to process image features from SigLIP.
Training is conducted on two primary objectives using English image-text data:
- Visually Conditioned Masked Language Modeling (VMLM): The model is trained to predict masked tokens in English captions by leveraging visual features. This forces the model to ground linguistic concepts in visual data.
- Kullback-Leibler (KL) Divergence Penalty: To prevent "catastrophic forgetting" of translation capabilities, the original text-only NLLB acts as a teacher. The MMT model is penalized if its output distribution for translation tasks deviates significantly from the teacher's predictions.
At inference, the framework utilizes Classifier-Free Guidance (CFG) to adjust the influence of visual cues. This allows for controllable multimodality, enabling the model to lean more heavily on images when text is ambiguous.
Experimental Setup
The model was trained on approximately 2.8 million English image-text pairs from the Conceptual Captions dataset. Evaluation was performed on the standard Multi30K benchmark and CoMMuTE, a contrastive benchmark for visual disambiguation. Furthermore, the authors extended CoMMuTE to include Arabic, Russian, and Chinese to test zero-shot generalization to languages without any multimodal training data.
Results and Discussion
ZeroMMT demonstrates disambiguation performance comparable to state-of-the-art supervised models. On the CoMMuTE English-to-French task, ZeroMMT achieved a score of 65.0, nearing the 67.1 score of the fully supervised VGAMT model. Crucially, the model maintained high translation fidelity, staying within ~0.5 BLEU points of the original NLLB on standard benchmarks, thanks to the KL penalty.
The extension to new languages confirmed that ZeroMMT generalizes effectively. In zero-shot scenarios for Arabic and Russian, the model successfully utilized visual context to improve translation accuracy, proving that multimodal capabilities can be learned from monolingual data and transferred across languages.
Conclusion
ZeroMMT provides a scalable path for extending MMT to low-resource languages by eliminating the requirement for parallel multimodal data. While the model occasionally suffers from visual "noise" in clear-text scenarios, future work involving ambiguity detection could further refine performance. The code, data, and models are publicly available to foster further research in zero-shot multimodal learning.
Summary updated