Abstract
Vision-language pre-trained models (VL-PTMs) like CLIP have demonstrated remarkable zero-shot capabilities, yet they remain largely English-centric. Expanding these models to support the world's diverse languages typically requires "Joint Learning" on massive multilingual datasets, which is computationally prohibitive and often impractical as new data emerges. This paper introduces CLL-CLIP, a framework designed for Continual Language Learning (CLL) that allows models to learn new languages sequentially without forgetting previously acquired knowledge. By utilizing an expandable token embedding layer and a novel Token Embedding Initialization and Regularization (TEIR) technique, the framework achieves state-of-the-art performance in multilingual image-text retrieval while maintaining high computational efficiency.
Introduction
The "diversity gap" in modern AI is particularly evident in vision-language models. While CLIP is a powerful tool for image-text understanding, its training data is predominantly English, limiting its utility for the global population. Ideally, we would train models on all 7,000+ world languages simultaneously; however, the computational cost of joint training is immense. Furthermore, in real-world scenarios, data for different languages often arrives at different times, necessitating a model that can learn incrementally.
Continual Language Learning (CLL) offers a solution but is plagued by Catastrophic Forgetting (CF). When a model is fine-tuned on a new language (e.g., French), its performance on previously learned languages (e.g., English) typically collapses. This research identifies two primary drivers of CF in this context: covariate shift, where new data distributions shock the model, and lexical overlap, where shared tokens across languages are updated in ways that overwrite previous knowledge.
The CLL-CLIP Framework
To address these challenges, the authors propose a streamlined architecture and a specialized regularization strategy.
1. Architecture: Expandable Token Embeddings
Rather than fine-tuning the entire transformer architecture, CLL-CLIP freezes the vision encoder and the majority of the text encoder. The model only trains an expandable token embedding layer. By keeping the core "reasoning" layers of the CLIP text encoder stable, the model preserves its fundamental understanding of semantics while allowing the vocabulary to grow as new languages are introduced. This approach ensures "memory stability" and significantly reduces the number of trainable parameters.
2. TEIR: Initialization and Regularization
The core innovation of this work is TEIR (Token Embedding Initialization and Regularization), which mitigates forgetting through two mechanisms:
- Distribution-Aware Initialization: Standard random initialization of new tokens can cause a sudden shift in the feature distribution, leading to forgetting. TEIR ensures that new token embeddings are initialized to follow the exact Gaussian distribution (mean and variance) of the previously learned embeddings, preventing distributional "shocks."
- Frequency-Based Regularization: To manage lexical overlap, TEIR applies a scaling factor to the learning process. It penalizes updates to tokens based on their frequency in previous tasks. If a token has appeared frequently in earlier languages, the model is discouraged from significantly altering its embedding, thereby protecting established knowledge.
Experimental Evaluation
The framework was evaluated using two primary benchmarks covering 36 languages:
- MSCOCO36: A benchmark created by translating MSCOCO captions into 35 additional languages.
- XM3600: A human-annotated, massively multilingual dataset used to test real-world generalization beyond machine translation.
Key Results
- Performance Gains: Integrating TEIR into the CLL-CLIP baseline resulted in a 6.7% increase in average Recall@1 for text-to-image retrieval on the XM3600 dataset.
- Superiority over Translation Pipelines: CLL-CLIP outperformed the "Translate-Test" method, where non-English queries are translated into English before being processed by a standard CLIP model. This suggests that native multilingual training is more effective than relying on intermediate translation.
- Compatibility: TEIR proved to be a versatile plug-and-play component, consistently improving the performance of other continual learning methods such as Experience Replay (ER) and LoRA.
- Efficiency: The model provides a scalable alternative to joint training, allowing for the addition of new languages with minimal computational overhead compared to retraining a full multilingual model from scratch.
Discussion and Limitations
A significant insight from this study is that training the entire transformer is not necessary for multilingual expansion; the token embeddings alone carry sufficient information to adapt the model to new languages. Furthermore, the use of simple token frequency counts for regularization provides an elegant, low-overhead alternative to complex gradient-based methods like Elastic Weight Consolidation (EWC).
However, some limitations remain. The TEIR regularization requires maintaining a running count of token frequencies, which may be challenging in "boundary-free" learning environments where task divisions are not clearly defined. Additionally, the current research focuses on dual-stream retrieval models; future work is needed to determine if these strategies apply to encoder-decoder models used for generative tasks like image captioning. Finally, the reliance on a fixed-size vocabulary for each task suggests that more fluid sub-word handling across thousands of languages remains an open area for exploration.
Conclusion
CLL-CLIP and the TEIR technique represent a significant step toward more inclusive and computationally sustainable vision-language models. By focusing on the statistical distribution and historical importance of token embeddings, the framework allows CLIP to embrace global linguistic diversity without sacrificing its original capabilities.
Summary updated