Abstract
Contrastive Language-Image Pre-training (CLIP) has revolutionized vision-language understanding but remains computationally intensive. While token masking can improve training efficiency, random masking often removes semantically critical information, introducing "semantic noise" that degrades model performance. We present Attentive Mask CLIP (A-CLIP), a novel training approach that utilizes an attentive masking strategy to preserve the most relevant image patches. By employing an Exponential Moving Average (EMA) teacher to guide mask selection, A-CLIP achieves superior accuracy with significantly reduced computational overhead. On ImageNet-1K zero-shot classification, A-CLIP reaches 43.9% accuracy, outperforming standard CLIP by 6.3% while offering up to a 2.3× speedup over previous state-of-the-art methods.
Introduction
The success of Vision Transformers (ViTs) in vision-language tasks is often bottlenecked by the massive scale of data and compute required. Token removal (masking) is a common efficiency technique, but its application in CLIP is challenging. Random masking can discard the very objects described in the paired text, creating a misalignment that hinders learning. A-CLIP addresses this by ensuring that the most semantically important patches are preserved during training, turning masking into a powerful form of data augmentation.
Methods
A-CLIP introduces a targeted masking approach to improve the alignment between visual tokens and textual descriptions.
- EMA Teacher and Attentive Masking: The framework maintains an Exponential Moving Average (EMA) version of the vision encoder. This teacher model calculates attention scores for image patches relative to the global
[CLS]token. Patches with the highest scores—representing the most semantically dense regions—are kept, while others are discarded. - Multi-View Contrastive Learning: The reduced token count allows for processing multiple masked "views" of an image within the same computational budget. Auxiliary self-supervised losses are applied across these views to enhance feature robustness.
- A-CLIP-eff: To further minimize overhead, this variant runs the EMA teacher at half-resolution, making the mask generation process highly efficient.
Experimental Setup
The models were trained on a 15-million subset of the YFCC-100M dataset (YFCC-15M). Evaluation spanned ImageNet-1K for zero-shot classification, Flickr30K and MS COCO for retrieval, and a suite of 25 downstream tasks including CIFAR, Food-101, and specialized datasets like EuroSAT.
Key Results
A-CLIP demonstrates a superior balance of speed and accuracy compared to existing methods like CLIP, SLIP, and MaskCLIP.
- Accuracy Gains: A-CLIP achieves 43.9% zero-shot accuracy on ImageNet-1K, significantly outperforming standard CLIP (37.6%) and the high-performing SLIP (42.8%).
- Training Efficiency: A-CLIP is 2.3× faster than SLIP. The "A-CLIP-eff" variant even surpasses the original CLIP's speed (0.86× training time) while maintaining a +5.3% accuracy lead.
- Retrieval Excellence: On Flickr30K, A-CLIP improved Image-to-Text retrieval by +11.3% over the baseline CLIP.
Discussion and Conclusion
The results suggest that masking is not merely a compression technique but a powerful form of data augmentation when guided by semantic importance. By focusing on high-attention tokens, A-CLIP ensures the model receives a clear signal for text alignment. While generating attentive masks adds some overhead, the overall throughput gains and accuracy improvements make it a highly effective strategy for large-scale vision-language pre-training. Future work may explore scaling A-CLIP to even larger datasets and integrating it with Masked Image Modeling (MIM) tasks.
Summary updated