Abstract
Existing vision-language methods typically support two languages at a time at most. In this paper, we present a modular approach which can easily be incorporated into existing vision-language methods in order to support many languages. We accomplish this by learning a single shared Multimodal Universal Language Embedding (MULE) which has been visually-semantically aligned across all languages. Then we learn to relate MULE to visual data as if it were a single language. Our method is not architecture specific, unlike prior work which typically learned separate branches for each language, enabling our approach to easily be adapted to many vision-language methods and tasks. Since MULE learns a single language branch in the multimodal model, we can also scale to support many languages, and languages with fewer annotations can take advantage of the good representation learned from other (more abundant) language data. We demonstrate the effectiveness of MULE on the bidirectional image-sentence retrieval task, supporting up to four languages in a single model. In addition, we show that Machine Translation can be used for data augmentation in multilingual learning, which, combined with MULE, improves mean recall by up to 21.9% on a single-language compared to prior work, with the most significant gains seen on languages with relatively few annotations. Our code is publicly available.
Introduction
Current vision-language models, particularly for tasks like image-to-sentence retrieval, are often "language-greedy," designed primarily for English or a maximum of two languages. Adding new languages typically requires adding separate model branches, leading to two significant issues: linear growth in model size (scalability) and the inability of low-resource languages (e.g., Czech or Chinese) to benefit from high-resource data like English (data scarcity). MULE addresses these by creating a universal language branch that treats all languages as one, facilitating parameter sharing and knowledge transfer.
Methodology
The core of our approach is the Multimodal Universal Language Embedding (MULE). Instead of separate encoders, MULE projects all languages into a single shared space. This space is aligned using three key components:
- Neighborhood Constraints: A triplet loss ensures that sentences in different languages describing the same image are mapped close together.
- Adversarial Language Classifier: An adversarial training setup forces the encoder to learn language-agnostic features by trying to "fool" a classifier that attempts to identify the source language.
- Image-Language Matching: Text embeddings are matched directly to visual features extracted from a CNN (e.g., ResNet-152), grounding the language in visual reality.
Additionally, we leverage Neural Machine Translation (NMT) for data augmentation, translating English captions into target languages to bolster training signals for data-poor languages.
Experimental Setup
We evaluated MULE on two benchmarks:
- Multi30K: Images with descriptions in English, German, French, and Czech.
- MSCOCO: Utilizing English, Japanese (YJ Captions), and Chinese (COCO-CN) descriptions.
These datasets exhibit significant language imbalance; for example, MSCOCO contains 606,000 English descriptions but only 20,000 in Chinese, providing a rigorous test for low-resource transfer learning.
Results and Discussion
MULE demonstrates superior performance, particularly on low-resource languages. On the Multi30K dataset, MULE improved mean recall for Czech by 21.9% compared to prior state-of-the-art methods. Furthermore, the shared branch architecture is highly efficient: a 4-language model on Multi30K requires only 13.6M parameters, half of the 27.2M required by traditional separate-branch models.
Key insights include:
- Visual Grounding: Visual information acts as a universal pivot, aligning linguistically distant languages through shared pixels.
- Multimodal vs. Text-only Alignment: Aligning languages using visual-semantic grounding significantly outperforms text-only alignment methods like MUSE.
- Augmentation Efficacy: Despite potential noise, NMT-augmented data provides a substantial boost, especially for Chinese and Japanese on MSCOCO.
Conclusion
While MULE offers massive scalability and benefits for low-resource languages, we observed a slight "English tax"—a minor performance drop in English when forced into a shared space. Future work will explore sophisticated sampling strategies to manage language imbalance and the application of MULE to tasks like Visual Question Answering (VQA) and image captioning.
Summary updated