A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene

Wenbo Zhang, Yifan Zhang, Jianfeng Lin, Binqiang Huang, Jinlu Zhang, Wenhao Yu

Abstract

Pre-trained vision-language (V-L) models such as CLIP have shown excellent performance in many downstream cross-modal tasks. However, most of them are only applicable to the English context. Subsequent research has focused on this problem and proposed improved models, such as CN-CLIP and AltCLIP, to facilitate their applicability to Chinese and even other languages. Nevertheless, these models suffer from high latency and a large memory footprint in inference, which limits their further deployment on resource-constrained edge devices. In this work, we propose a conceptually simple yet effective multilingual CLIP Compression framework and train a lightweight multilingual vision-language model, called DC-CLIP, for both Chinese and English context. In this framework, we collect high-quality Chinese and English text-image pairs and design two training stages, including multilingual vision-language feature distillation and alignment. During the first stage, lightweight image/text student models are designed to learn robust visual/multilingual textual feature representation ability from corresponding teacher models, respectively. Subsequently, the multilingual vision-language alignment stage enables effective alignment of visual and multilingual textual features to further improve the model's multilingual performance. Comprehensive experiments in zero-shot image classification, conducted based on the ELEVATER benchmark, showcase that DC-CLIP achieves superior performance in the English context and competitive performance in the Chinese context, even with less training data, when compared to existing models of similar parameter magnitude. The evaluation demonstrates the effectiveness of our designed training mechanism.

Introduction

Large-scale Vision-Language (V-L) models, most notably OpenAI’s CLIP, have revolutionized how AI understands the relationship between images and text. However, the original CLIP has two major "blind spots": it is primarily English-centric, and it is computationally massive. While subsequent models like AltCLIP and CN-CLIP introduced multilingual capabilities, they inherited (or even increased) the high latency and large memory requirements of their predecessors. This makes them nearly impossible to deploy on resource-constrained "edge" devices like smartphones or IoT sensors. The challenge addressed in this work is creating a model small enough for edge deployment yet capable of understanding multiple languages and complex visual scenes.

Methodology

The proposed framework, DC-CLIP, utilizes a "Progressive Framework" consisting of two distinct stages: Vision-Language Feature Distillation and Vision-Language Feature Alignment.

Vision-Language Feature Distillation

Instead of training from scratch, a "Teacher-Student" approach is employed. The massive AltCLIP model acts as the teacher, transferring knowledge to a smaller student model. The student architecture consists of a ResNet50 for images and a mini-model derived from XLM-R for text. Unlike traditional distillation that focuses on final logits, this framework performs feature-based distillation using a Smooth L1 loss. This forces the student’s internal feature maps to match the teacher’s, ensuring the student captures nuanced representations.

Vision-Language Feature Alignment

Distillation alone is insufficient to ensure image and text encoders are perfectly aligned. In the second stage, contrastive learning (InfoNCE loss) is applied using a high-quality bilingual dataset. During this stage, the image encoder is frozen, and only the text encoder is updated. This stabilizes training and focuses the model on aligning multilingual concepts to a fixed visual space.

The two-stage progressive training framework: (a) Feature Distillation and (b) Feature Alignment.

Experimental Results

DC-CLIP (301M parameters) was evaluated against other "tiny" variants like CN-CLIP and TaiyiCLIP using the ELEVATER benchmark, which includes eight diverse datasets ranging from generic objects to satellite imagery.

Performance Analysis

In English zero-shot classification, DC-CLIP significantly outperformed its peers. On ImageNet, it achieved 42.79% accuracy, compared to CN-CLIP’s 11.04% and TaiyiCLIP’s 14.84%. In the Chinese context, it remained highly competitive, outperforming ChineseCLIP across all assessed datasets and beating TaiyiCLIP in four out of eight categories.

Zero-shot image classification performance comparison on the ELEVATER benchmark.

Key Insights

  • Feature Maps over Logits: Output feature maps provide a richer knowledge transfer than simple probability vectors, allowing the student to learn the teacher's visual perception.
  • Efficiency with Limited Data: DC-CLIP achieves high performance using only ~3 million image-text pairs, a fraction of the data used by models like TinyCLIP.
  • Hardware Friendly: The model is approximately 1/10th the size of AltCLIP, making it viable for mobile deployment without the massive latency issues of larger models.

Conclusion

DC-CLIP represents a significant step toward bringing multilingual vision-language capabilities to edge devices. While it shows state-of-the-art performance in English, there remains a performance gap in Chinese due to data quality. Future work will focus on extending this distillation process to more languages and diverse domains, alongside further optimizing feature mapping techniques.

Summary updated

Abstract

Pre-trained vision-language (V-L) models such as CLIP have shown excellent performance in many downstream cross-modal tasks. However, most of them are only applicable to the English context. Subsequent research has focused on this problem and proposed improved models, such as CN-CLIP and AltCLIP, to facilitate their applicability to Chinese and even other languages. Nevertheless, these models suffer from high latency and a large memory footprint in inference, which limits their further deployment on resource-constrained edge devices. In this work, we propose a conceptually simple yet effective multilingual CLIP Compression framework and train a lightweight multilingual vision-language model, called DC-CLIP, for both Chinese and English context. In this framework, we collect high-quality Chinese and English text-image pairs and design two training stages, including multilingual vision-language feature distillation and alignment. During the first stage, lightweight image/text student models are designed to learn robust visual/multilingual textual feature representation ability from corresponding teacher models, respectively. Subsequently, the multilingual vision-language alignment stage enables effective alignment of visual and multilingual textual features to further improve the model's multilingual performance. Comprehensive experiments in zero-shot image classification, conducted based on the ELEVATER benchmark, showcase that DC-CLIP achieves superior performance in the English context and competitive performance in the Chinese context, even with less training data, when compared to existing models of similar parameter magnitude. The evaluation demonstrates the effectiveness of our designed training mechanism.
BibTeX
@misc{zhang2024progressive,
  author = {Wenbo Zhang and Yifan Zhang and Jianfeng Lin and Binqiang Huang and Jinlu Zhang and Wenhao Yu},
  title = {{A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene}},
  year = {2024},
  eprint = {2404.11249},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.11249},
}
Figure