Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning

Bang Yang, Yong Dai, Xuxin Cheng, Yaowei Li, Asif Raza, Yuexian Zou

Abstract

Vision-language pre-trained models (VL-PTMs) like CLIP have demonstrated remarkable zero-shot capabilities, yet they remain largely English-centric. Expanding these models to support the world's diverse languages typically requires "Joint Learning" on massive multilingual datasets, which is computationally prohibitive and often impractical as new data emerges. This paper introduces CLL-CLIP, a framework designed for Continual Language Learning (CLL) that allows models to learn new languages sequentially without forgetting previously acquired knowledge. By utilizing an expandable token embedding layer and a novel Token Embedding Initialization and Regularization (TEIR) technique, the framework achieves state-of-the-art performance in multilingual image-text retrieval while maintaining high computational efficiency.

Introduction

The "diversity gap" in modern AI is particularly evident in vision-language models. While CLIP is a powerful tool for image-text understanding, its training data is predominantly English, limiting its utility for the global population. Ideally, we would train models on all 7,000+ world languages simultaneously; however, the computational cost of joint training is immense. Furthermore, in real-world scenarios, data for different languages often arrives at different times, necessitating a model that can learn incrementally.

Continual Language Learning (CLL) offers a solution but is plagued by Catastrophic Forgetting (CF). When a model is fine-tuned on a new language (e.g., French), its performance on previously learned languages (e.g., English) typically collapses. This research identifies two primary drivers of CF in this context: covariate shift, where new data distributions shock the model, and lexical overlap, where shared tokens across languages are updated in ways that overwrite previous knowledge.

The CLL-CLIP Framework

To address these challenges, the authors propose a streamlined architecture and a specialized regularization strategy.

1. Architecture: Expandable Token Embeddings

Rather than fine-tuning the entire transformer architecture, CLL-CLIP freezes the vision encoder and the majority of the text encoder. The model only trains an expandable token embedding layer. By keeping the core "reasoning" layers of the CLIP text encoder stable, the model preserves its fundamental understanding of semantics while allowing the vocabulary to grow as new languages are introduced. This approach ensures "memory stability" and significantly reduces the number of trainable parameters.

2. TEIR: Initialization and Regularization

The core innovation of this work is TEIR (Token Embedding Initialization and Regularization), which mitigates forgetting through two mechanisms:

  • Distribution-Aware Initialization: Standard random initialization of new tokens can cause a sudden shift in the feature distribution, leading to forgetting. TEIR ensures that new token embeddings are initialized to follow the exact Gaussian distribution (mean and variance) of the previously learned embeddings, preventing distributional "shocks."
  • Frequency-Based Regularization: To manage lexical overlap, TEIR applies a scaling factor to the learning process. It penalizes updates to tokens based on their frequency in previous tasks. If a token has appeared frequently in earlier languages, the model is discouraged from significantly altering its embedding, thereby protecting established knowledge.

Experimental Evaluation

The framework was evaluated using two primary benchmarks covering 36 languages:

  • MSCOCO36: A benchmark created by translating MSCOCO captions into 35 additional languages.
  • XM3600: A human-annotated, massively multilingual dataset used to test real-world generalization beyond machine translation.

Key Results

  • Performance Gains: Integrating TEIR into the CLL-CLIP baseline resulted in a 6.7% increase in average Recall@1 for text-to-image retrieval on the XM3600 dataset.
  • Superiority over Translation Pipelines: CLL-CLIP outperformed the "Translate-Test" method, where non-English queries are translated into English before being processed by a standard CLIP model. This suggests that native multilingual training is more effective than relying on intermediate translation.
  • Compatibility: TEIR proved to be a versatile plug-and-play component, consistently improving the performance of other continual learning methods such as Experience Replay (ER) and LoRA.
  • Efficiency: The model provides a scalable alternative to joint training, allowing for the addition of new languages with minimal computational overhead compared to retraining a full multilingual model from scratch.

Discussion and Limitations

A significant insight from this study is that training the entire transformer is not necessary for multilingual expansion; the token embeddings alone carry sufficient information to adapt the model to new languages. Furthermore, the use of simple token frequency counts for regularization provides an elegant, low-overhead alternative to complex gradient-based methods like Elastic Weight Consolidation (EWC).

However, some limitations remain. The TEIR regularization requires maintaining a running count of token frequencies, which may be challenging in "boundary-free" learning environments where task divisions are not clearly defined. Additionally, the current research focuses on dual-stream retrieval models; future work is needed to determine if these strategies apply to encoder-decoder models used for generative tasks like image captioning. Finally, the reliance on a fixed-size vocabulary for each task suggests that more fluid sub-word handling across thousands of languages remains an open area for exploration.

Conclusion

CLL-CLIP and the TEIR technique represent a significant step toward more inclusive and computationally sustainable vision-language models. By focusing on the statistical distribution and historical importance of token embeddings, the framework allows CLIP to embrace global linguistic diversity without sacrificing its original capabilities.

Summary updated

Abstract

While vision-language pre-trained models (VL-PTMs) have advanced multimodal research in recent years, their mastery in a few languages like English restricts their applicability in broader communities. To this end, there is an increasing interest in developing multilingual VL models via a joint-learning setup, which, however, could be unrealistic due to expensive costs and data availability. In this work, we propose to extend VL-PTMs' language capacity by continual language learning (CLL), where a model needs to update its linguistic knowledge incrementally without suffering from catastrophic forgetting (CF). We begin our study by introducing a model dubbed CLL-CLIP, which builds upon CLIP, a prevailing VL-PTM that has acquired image-English text alignment. Specifically, CLL-CLIP contains an expandable token embedding layer to handle linguistic differences. It solely trains token embeddings to improve memory stability and is optimized under cross-modal and cross-lingual objectives to learn the alignment between images and multilingual texts. To alleviate CF raised by covariate shift and lexical overlap, we further propose a novel approach that ensures the identical distribution of all token embeddings during initialization and regularizes token embedding learning during training. We construct a CLL benchmark covering 36 languages based on MSCOCO and XM3600 datasets and then evaluate multilingual image-text retrieval performance. Extensive experiments verify the effectiveness of CLL-CLIP and show that our approach can boost CLL-CLIP, e.g., by 6.7% in text-to-image average Recall@1 on XM3600, and improve various state-of-the-art methods consistently. Our code and data are available at https://github.com/yangbang18/CLFM.
BibTeX
@inproceedings{yang2024embracing,
  author = {Bang Yang and Yong Dai and Xuxin Cheng and Yaowei Li and Asif Raza and Yuexian Zou},
  title = {{Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=0Ycqxx0WZn},
}
Figure