CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance

Chu Myaet Thwal, Ye Lin Tun, Minh N. H. Nguyen, Eui-Nam Huh, Choong Seon HongarXiv 2024 · DOI · arXiv

Abstract

Contrastive Language-Image Pre-training (CLIP) models are typically large and data-intensive, posing challenges for deployment on resource-constrained devices. We present CLIP-PING (Proximus Intrinsic Neighbors Guidance), an efficient training paradigm that enhances lightweight vision-language models. By utilizing frozen features from pre-trained unimodal models to identify semantic neighbors, CLIP-PING provides additional contrastive supervision without the heavy computational overhead of traditional knowledge distillation. Our approach achieves significant performance gains in zero-shot classification and cross-modal retrieval while maintaining a memory footprint comparable to vanilla CLIP.

Introduction

The success of CLIP has transformed computer vision, yet the massive scale of these models and their reliance on enormous datasets limit their utility for edge computing. Training lightweight versions often results in poor cross-modal alignment when data is scarce. While Knowledge Distillation (KD) can bridge this gap, it typically requires running a large teacher model concurrently, leading to prohibitive GPU memory and time costs. CLIP-PING addresses this by asking: Can we transfer knowledge from large pre-trained models to lightweight students without the computational burden of active distillation?

Methodology

CLIP-PING introduces a two-stage process designed for maximum efficiency:

  1. Unimodal Feature Extraction: Pre-trained, off-the-shelf unimodal models (e.g., ResNet-50, BERT) extract features from the training set once. These are stored in offline "auxiliary feature banks," allowing the teacher models to be discarded before student training begins.
  2. Multi-Modal Training with PING: During training of lightweight encoders (e.g., ViT-XS, MobileBERT), the system retrieves neighbors from these banks to apply two novel losses:
  3. Intra-modal Nearest-Neighbor (NN) Supervision: Aligns student representations with the semantic groupings established by the teacher within a single modality.
  4. Inter-modal Cross Nearest-Neighbor (XNN) Supervision: Facilitates indirect cross-modal alignment. If an image is paired with a specific text, and the teacher identifies a second text as a near-neighbor to the first, the student is encouraged to align the image with both.
The CLIP-PING training pipeline, featuring frozen unimodal feature extractors and the PING supervision mechanism.

An active variant, A-CLIP-PING, is also proposed for scenarios where higher compute is available, keeping teachers active for dynamic guidance.

Experimental Setup

The framework was evaluated on relatively small datasets to simulate resource-constrained environments:

  • Training: COCO (600K pairs) and COCO + CC3M (3M pairs).
  • Evaluation: Zero-shot classification (ImageNet-1K, CIFAR, etc.), cross-modal retrieval (Flickr30K), and robustness benchmarks.
  • Architectures: ViT-XS, ConvNeXt-Pico, and MobileNetV4-Hybrid.

Key Results and Efficiency

CLIP-PING demonstrates that implicit guidance via neighbor relationships is highly effective. On ImageNet-1K, a ViT-XS model trained with CLIP-PING achieved a +5.5% zero-shot accuracy improvement over vanilla CLIP. In retrieval tasks on Flickr30K, it saw gains of +10.7% (I2T) and +5.7% (T2I).

Performance comparison across multiple benchmarks (left) and training memory usage (right) for CLIP-PING vs. baselines.

Crucially, CLIP-PING is 1.3× to 1.8× faster than standard CLIP-KD. By decoupling teacher inference from student training, it maintains a computational profile nearly identical to unguided training while delivering the performance benefits of distillation.

Conclusion

CLIP-PING proves that semantic richness from large models can be distilled into lightweight architectures through simple neighbor-based supervision. This approach effectively mitigates the "false negative" problem in standard contrastive learning by recognizing that multiple samples can share the same semantic space. Future work will explore extending PING to other modalities like audio and video for on-device applications.

Summary updated

Abstract

Beyond the success of Contrastive Language-Image Pre-training (CLIP), recent trends mark a shift toward exploring the applicability of lightweight vision-language models for resource-constrained scenarios. These models often deliver suboptimal performance when relying solely on a single image-text contrastive learning objective, spotlighting the need for more effective training mechanisms that guarantee robust cross-modal feature alignment. In this work, we propose CLIP-PING: Contrastive Language-Image Pre-training with Proximus Intrinsic Neighbors Guidance, a novel yet simple and efficient training paradigm designed to boost the performance of lightweight vision-language models with minimal computational overhead and lower data demands. CLIP-PING bootstraps unimodal features extracted from arbitrary pre-trained encoders to obtain intrinsic guidance of proximus neighbor samples, i.e., nearest-neighbor (NN) and cross nearest-neighbor (XNN). We find that extra contrastive supervision from these neighbors substantially boosts cross-modal alignment, enabling lightweight models to learn more generic features with rich semantic diversity. Extensive experiments reveal that CLIP-PING notably surpasses its peers in zero-shot generalization and cross-modal retrieval tasks. Specifically, a 5.5% gain on zero-shot ImageNet1K classification with 10.7% (I2T) and 5.7% (T2I) on Flickr30K retrieval, compared to the original CLIP when using ViT-XS image encoder trained on 3 million (image, text) pairs. Moreover, CLIP-PING showcases a strong transferability under the linear evaluation protocol across several downstream tasks.
BibTeX
@misc{thwal2024clip,
  author = {Chu Myaet Thwal and Ye Lin Tun and Minh N. H. Nguyen and Eui-Nam Huh and Choong Seon Hong},
  title = {{CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance}},
  year = {2024},
  eprint = {2412.03871},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.03871},
}
Figure