MULE: Multimodal Universal Language Embedding

Donghyun Kim, Kuniaki Saito, Kate Saenko, Stan Sclaroff, Bryan A. Plummer

Abstract

Existing vision-language methods typically support two languages at a time at most. In this paper, we present a modular approach which can easily be incorporated into existing vision-language methods in order to support many languages. We accomplish this by learning a single shared Multimodal Universal Language Embedding (MULE) which has been visually-semantically aligned across all languages. Then we learn to relate MULE to visual data as if it were a single language. Our method is not architecture specific, unlike prior work which typically learned separate branches for each language, enabling our approach to easily be adapted to many vision-language methods and tasks. Since MULE learns a single language branch in the multimodal model, we can also scale to support many languages, and languages with fewer annotations can take advantage of the good representation learned from other (more abundant) language data. We demonstrate the effectiveness of our embeddings on the bidirectional image-sentence retrieval task, supporting up to four languages in a single model. In addition, we show that Machine Translation can be used for data augmentation in multilingual learning, which, combined with MULE, improves mean recall by up to 20.2% on a single language compared to prior work, with the most significant gains seen on languages with relatively few annotations.

Introduction: Bridging the Language Gap in Vision-Language AI

Most current AI models that connect images and text are "monolingual"—they usually only support English. When researchers attempt to add more languages, they typically encounter two significant barriers:

  1. The Parameter Explosion: Standard methods create a separate "branch" for every new language. Supporting 10 languages makes the model ten times larger and significantly harder to manage.
  2. The Data Desert: While massive datasets exist for English image captions (like MSCOCO), there are very few for languages like Japanese, Chinese, or Czech. Models trained only on these small datasets perform poorly.
MULE vs. Existing Methods: MULE uses a shared universal embedding space instead of separate language branches, allowing for better scalability and cross-lingual knowledge transfer.

The MULE (Multimodal Universal Language Embedding) framework addresses these issues by creating a system that is both scalable (adding languages does not exponentially increase parameters) and collaborative (knowledge from high-resource languages like English helps the model understand low-resource languages).

Methods: The MULE Framework

The core innovation is the shared Multimodal Universal Language Embedding. Instead of separate pipelines, every language is projected into a single shared space through a lightweight, language-specific layer. This allows the model to relate MULE to visual data as if it were a single language.

Detailed MULE Architecture: The model aligns language-specific word embeddings into a shared MULE space, which is then processed by a shared multimodal model and aligned with visual features using three distinct loss constraints.

To ensure this shared space is effective, the authors employ a three-pronged training strategy:

  • Neighborhood Constraints: A triplet loss function forces sentences describing the same image—regardless of the language—to be close to each other in the embedding space.
  • Adversarial Language Classifier: A "critic" tries to guess which language a piece of text belongs to. The model is trained to "fool" this critic, forcing the features of different languages to have nearly identical distributions.
  • Image-Sentence Matching: A shared multimodal branch (using an LSTM and fully connected layers) learns to relate this universal language space to visual features extracted from a CNN.

Additionally, the researchers utilized Neural Machine Translation (NMT) as a data augmentation tool, translating English captions into other languages to provide more training examples for data-scarce languages.

Key Insights and Results

  • Images as a Universal Pivot: Visual data acts as a universal anchor. Because a "dog" looks the same regardless of the language used to describe it, visual context provides the "glue" that aligns languages more effectively than text-only methods.
  • Massive Gains for Low-Resource Languages: On the Multi30K dataset, MULE improved the mean recall (mR) for Czech by 20.2% compared to prior work. On MSCOCO, it improved Chinese performance by 5.9% while maintaining competitive performance on English.
  • Efficiency and Scalability: MULE is remarkably parameter-efficient. For a four-language model, it uses roughly half the parameters of previous state-of-the-art "parallel" models because it eliminates the need for separate branches for every language.
  • Outperforming Text-Only Alignments: MULE significantly outperformed state-of-the-art multilingual embeddings like MUSE and Multilingual BERT, proving that visual grounding is essential for vision-language tasks.

Limitations and Future Work

The authors noted a slight "performance tax" on the primary high-resource language (English) when training in a multilingual setting, likely due to the massive imbalance in training data (e.g., 606K English descriptions vs. 20K Chinese descriptions). Future work may involve developing more sophisticated sampling strategies to balance these datasets. Furthermore, the modular nature of MULE allows it to be potentially adapted to other tasks like Visual Question Answering (VQA) or image captioning.

Summary updated

Abstract

Existing vision-language methods typically support two languages at a time at most. In this paper, we present a modular approach which can easily be incorporated into existing vision-language methods in order to support many languages. We accomplish this by learning a single shared Multimodal Universal Language Embedding (MULE) which has been visually-semantically aligned across all languages. Then we learn to relate MULE to visual data as if it were a single language. Our method is not architecture specific, unlike prior work which typically learned separate branches for each language, enabling our approach to easily be adapted to many vision-language methods and tasks. Since MULE learns a single language branch in the multimodal model, we can also scale to support many languages, and languages with fewer annotations can take advantage of the good representation learned from other (more abundant) language data. We demonstrate the effectiveness of our embeddings on the bidirectional image-sentence retrieval task, supporting up to four languages in a single model. In addition, we show that Machine Translation can be used for data augmentation in multilingual learning, which, combined with MULE, improves mean recall by up to 20.2% on a single language compared to prior work, with the most significant gains seen on languages with relatively few annotations. Our code is publicly available1.
BibTeX
@inproceedings{kim2020mule,
  author = {Donghyun Kim and Kuniaki Saito and Kate Saenko and Stan Sclaroff and Bryan A. Plummer},
  title = {{MULE: Multimodal Universal Language Embedding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=gLyGRGH4bt},
}
Figure