TL;DR
This paper introduces Multi-Lingual Acquisition (MLA), a highly efficient framework that retrofits existing monolingual Vision-Language Pre-training (VLP) models (like CLIP) with multilingual capabilities. By using lightweight "language acquirers" and a two-stage training strategy inspired by human language learning, MLA achieves state-of-the-art cross-modal retrieval across multiple languages using a fraction of the data and compute required by massive multilingual models.
What Problem Does This Paper Address?
To build AI that understands both visual and textual information across the globe, researchers typically train Multilingual Vision-Language Pre-training (M-VLP) models from scratch. Models like MURAL or UC$^2$ process images and dozens of languages simultaneously during pre-training.
However, this approach has two major flaws:
- Astronomical Compute Costs: Training these models requires massive datasets and hardware (e.g., MURAL requires 128 TPUv3s running for 4 days).
- Inflexibility & Catastrophic Forgetting: If you want to add a new language to an existing M-VLP, you have to retrain it. Because model capacity is limited, forcing it to learn a new language often degrades its performance on the original languages (including English).
The authors ask: Instead of training a massive multilingual model from scratch, can we take a powerful, pre-trained monolingual model (like English CLIP) and teach it new languages cheaply and modularly?
Methods
The authors propose the Multi-Lingual Acquisition (MLA) framework. Instead of updating the core VLP model, they freeze the pre-trained vision and text encoders and inject lightweight, language-specific modules.
- Language Acquirers: For each new language, MLA adds a shared non-native embedding block and a tiny "Language Acquirer" (a bottleneck MLP with residual connections) into the frozen text encoder. Each acquirer is independent and costs only ~3.14 MB of parameters.
- Two-Stage Training Strategy: Inspired by how humans learn a second language, the authors optimize these acquirers in two phases:
- Native Language Transfer (NLT): When humans learn a new language, they first translate it to their mother tongue. In this stage, MLA uses translation pairs (e.g., French-to-English) and a Mean Square Error (MSE) loss to force the non-native text representation to align with the frozen English text representation.
- Language Exposure (LE): Once a foundation is built, humans practice by interacting with the real world. In this stage, MLA uses image-text pairs and contrastive learning to align the non-native text representations directly with the frozen image representations.
Datasets
- Training: The model is trained on subsets of Conceptual Captions (CC). To prove data efficiency, they primarily use a small CC300K subset. For multilingual data, they use machine-translated versions of CC: CC6L (5 translated languages) for image-text, and CC69L (68 translated languages) for video-text.
- Evaluation (Image-Text): Multi30K, MSCOCO (with Japanese and Chinese extensions), and XTD (11 languages for zero-shot evaluation).
- Evaluation (Video-Text): MSRVTT (translated into 8 languages).
Key Insights and Ideas
- Decoupling prevents forgetting: Because each language gets its own independent 3.14 MB acquirer, adding a new language has absolutely zero impact on the model's performance in English or any other previously learned language.
- Human-like curriculum learning works: Ablation studies show that the order of training matters immensely. If the model tries to align a new language directly with images (LE stage) without first aligning it to English (NLT stage), performance drops significantly.
- Extreme Data Efficiency: MLA proves that you don't need millions of image-text pairs to teach a vision model a new language. By piggybacking on CLIP's existing English-to-Vision alignment, MLA can learn a new language using as few as 300K translation pairs.
Key Results
- Compute Savings: MLA can be trained on a single Nvidia V100 GPU in about 12 hours. This is orders of magnitude cheaper than standard M-VLP pre-training.
- Zero-Shot Image-Text Retrieval: Using only 300K training pairs, $\text{MLA}_{\text{CLIP}}$ drastically outperforms the SOTA MURAL model trained on the same data (e.g., 78.7 vs 62.7 Average Recall on German). It even outperforms MURAL models trained on 12 million pairs.
- Video-Text Retrieval: Despite being trained only on static images (CC69L), $\text{MLA}_{\text{CLIP}}$ generalizes beautifully to video, beating the SOTA video-specific model XLM-R-MMP on MSRVTT across multiple languages.
- Low-Resource Viability: The authors demonstrated that MLA can achieve competitive cross-modal retrieval for a new language (like Korean) using only text-to-text translation pairs, without needing a single image-text pair in that language.
Limitations and Future Work
- Reliance on Bilingual Corpora: The NLT stage requires text-to-text translation pairs. This limits MLA's applicability to truly low-resource languages where bilingual dictionaries or translation datasets don't exist. Future work will explore using strictly monolingual corpora.
- Task Scope: The paper currently only evaluates MLA on coarse-grained cross-modal retrieval. It remains to be seen how well this lightweight acquisition translates to fine-grained tasks like Visual Question Answering (VQA).
- Translation Bias: Because the training data relies heavily on commercial machine translation services, any inherent biases or errors in those translation engines will likely be inherited by the MLA acquirers.
Summary updated