Abstract
Contrastive Language-Image Pre-training (CLIP) models are typically large and data-intensive, posing challenges for deployment on resource-constrained devices. We present CLIP-PING (Proximus Intrinsic Neighbors Guidance), an efficient training paradigm that enhances lightweight vision-language models. By utilizing frozen features from pre-trained unimodal models to identify semantic neighbors, CLIP-PING provides additional contrastive supervision without the heavy computational overhead of traditional knowledge distillation. Our approach achieves significant performance gains in zero-shot classification and cross-modal retrieval while maintaining a memory footprint comparable to vanilla CLIP.
Introduction
The success of CLIP has transformed computer vision, yet the massive scale of these models and their reliance on enormous datasets limit their utility for edge computing. Training lightweight versions often results in poor cross-modal alignment when data is scarce. While Knowledge Distillation (KD) can bridge this gap, it typically requires running a large teacher model concurrently, leading to prohibitive GPU memory and time costs. CLIP-PING addresses this by asking: Can we transfer knowledge from large pre-trained models to lightweight students without the computational burden of active distillation?
Methodology
CLIP-PING introduces a two-stage process designed for maximum efficiency:
- Unimodal Feature Extraction: Pre-trained, off-the-shelf unimodal models (e.g., ResNet-50, BERT) extract features from the training set once. These are stored in offline "auxiliary feature banks," allowing the teacher models to be discarded before student training begins.
- Multi-Modal Training with PING: During training of lightweight encoders (e.g., ViT-XS, MobileBERT), the system retrieves neighbors from these banks to apply two novel losses:
- Intra-modal Nearest-Neighbor (NN) Supervision: Aligns student representations with the semantic groupings established by the teacher within a single modality.
- Inter-modal Cross Nearest-Neighbor (XNN) Supervision: Facilitates indirect cross-modal alignment. If an image is paired with a specific text, and the teacher identifies a second text as a near-neighbor to the first, the student is encouraged to align the image with both.
An active variant, A-CLIP-PING, is also proposed for scenarios where higher compute is available, keeping teachers active for dynamic guidance.
Experimental Setup
The framework was evaluated on relatively small datasets to simulate resource-constrained environments:
- Training: COCO (600K pairs) and COCO + CC3M (3M pairs).
- Evaluation: Zero-shot classification (ImageNet-1K, CIFAR, etc.), cross-modal retrieval (Flickr30K), and robustness benchmarks.
- Architectures: ViT-XS, ConvNeXt-Pico, and MobileNetV4-Hybrid.
Key Results and Efficiency
CLIP-PING demonstrates that implicit guidance via neighbor relationships is highly effective. On ImageNet-1K, a ViT-XS model trained with CLIP-PING achieved a +5.5% zero-shot accuracy improvement over vanilla CLIP. In retrieval tasks on Flickr30K, it saw gains of +10.7% (I2T) and +5.7% (T2I).
Crucially, CLIP-PING is 1.3× to 1.8× faster than standard CLIP-KD. By decoupling teacher inference from student training, it maintains a computational profile nearly identical to unguided training while delivering the performance benefits of distillation.
Conclusion
CLIP-PING proves that semantic richness from large models can be distilled into lightweight architectures through simple neighbor-based supervision. This approach effectively mitigates the "false negative" problem in standard contrastive learning by recognizing that multiple samples can share the same semantic space. Future work will explore extending PING to other modalities like audio and video for on-device applications.
Summary updated