Attentive Mask CLIP

Yifan Yang, Weiquan Huang, Yixuan Wei, Houwen Peng, Xinyang Jiang, Huiqiang Jiang, Fangyun Wei, Yin Wang, Han Hu, Lili Qiu, Yuqing Yang

Abstract

Contrastive Language-Image Pre-training (CLIP) has revolutionized vision-language understanding but remains computationally intensive. While token masking can improve training efficiency, random masking often removes semantically critical information, introducing "semantic noise" that degrades model performance. We present Attentive Mask CLIP (A-CLIP), a novel training approach that utilizes an attentive masking strategy to preserve the most relevant image patches. By employing an Exponential Moving Average (EMA) teacher to guide mask selection, A-CLIP achieves superior accuracy with significantly reduced computational overhead. On ImageNet-1K zero-shot classification, A-CLIP reaches 43.9% accuracy, outperforming standard CLIP by 6.3% while offering up to a 2.3× speedup over previous state-of-the-art methods.

Introduction

The success of Vision Transformers (ViTs) in vision-language tasks is often bottlenecked by the massive scale of data and compute required. Token removal (masking) is a common efficiency technique, but its application in CLIP is challenging. Random masking can discard the very objects described in the paired text, creating a misalignment that hinders learning. A-CLIP addresses this by ensuring that the most semantically important patches are preserved during training, turning masking into a powerful form of data augmentation.

Methods

A-CLIP introduces a targeted masking approach to improve the alignment between visual tokens and textual descriptions.

  1. EMA Teacher and Attentive Masking: The framework maintains an Exponential Moving Average (EMA) version of the vision encoder. This teacher model calculates attention scores for image patches relative to the global [CLS] token. Patches with the highest scores—representing the most semantically dense regions—are kept, while others are discarded.
  2. Multi-View Contrastive Learning: The reduced token count allows for processing multiple masked "views" of an image within the same computational budget. Auxiliary self-supervised losses are applied across these views to enhance feature robustness.
  3. A-CLIP-eff: To further minimize overhead, this variant runs the EMA teacher at half-resolution, making the mask generation process highly efficient.
Comparison between Attentive Masking and Random Masking. Attentive masking preserves the core semantic content (the car), whereas random masking may leave only background elements, creating semantic noise.

Experimental Setup

The models were trained on a 15-million subset of the YFCC-100M dataset (YFCC-15M). Evaluation spanned ImageNet-1K for zero-shot classification, Flickr30K and MS COCO for retrieval, and a suite of 25 downstream tasks including CIFAR, Food-101, and specialized datasets like EuroSAT.

Key Results

A-CLIP demonstrates a superior balance of speed and accuracy compared to existing methods like CLIP, SLIP, and MaskCLIP.

  • Accuracy Gains: A-CLIP achieves 43.9% zero-shot accuracy on ImageNet-1K, significantly outperforming standard CLIP (37.6%) and the high-performing SLIP (42.8%).
  • Training Efficiency: A-CLIP is 2.3× faster than SLIP. The "A-CLIP-eff" variant even surpasses the original CLIP's speed (0.86× training time) while maintaining a +5.3% accuracy lead.
  • Retrieval Excellence: On Flickr30K, A-CLIP improved Image-to-Text retrieval by +11.3% over the baseline CLIP.
Performance and efficiency comparison of A-CLIP against CLIP, SLIP, and MaskCLIP. A-CLIP achieves the best accuracy-efficiency trade-off across multiple benchmarks.

Discussion and Conclusion

The results suggest that masking is not merely a compression technique but a powerful form of data augmentation when guided by semantic importance. By focusing on high-attention tokens, A-CLIP ensures the model receives a clear signal for text alignment. While generating attentive masks adds some overhead, the overall throughput gains and accuracy improvements make it a highly effective strategy for large-scale vision-language pre-training. Future work may explore scaling A-CLIP to even larger datasets and integrating it with Masked Image Modeling (MIM) tasks.

Summary updated

Abstract

Image token removal is an efficient augmentation strategy for reducing the cost of computing image features. However, this efficient augmentation strategy has been found to adversely affect the accuracy of CLIP-based training. We hypothesize that removing a large portion of image tokens may improperly discard the semantic content associated with a given text description, thus constituting an incorrect pairing target in CLIP training. To address this issue, we propose an attentive token removal approach for CLIP training, which retains tokens with a high semantic correlation to the text description. The correlation scores are computed in an online fashion using the EMA version of the visual encoder. Our experiments show that the proposed attentive masking approach performs better than the previous method of random token removal for CLIP training. The approach also makes it efficient to apply multiple augmentation views to the image, as well as introducing instance contrastive learning tasks between these views into the CLIP framework. Compared to other CLIP improvements that combine different pre-training targets such as SLIP and MaskCLIP, our method is not only more effective, but also much more efficient. Specifically, using ViT-B and YFCC-15M dataset, our approach achieves $43.9\%$ top-1 accuracy on ImageNet-1K zero-shot classification, as well as $62.7/42.1$ and $38.0/23.2$ I2T/T2I retrieval accuracy on Flickr30K and MS COCO, which are $+1.1\%$, $+5.5/+0.9$, and $+4.4/+1.3$ higher than the SLIP method, while being $2.30\times$ faster. An efficient version of our approach running $1.16\times$ faster than the plain CLIP model achieves significant gains of $+5.3\%$, $+11.3/+8.0$, and $+9.5/+4.9$ on these benchmarks.
BibTeX
@inproceedings{yang2022attentive,
  author = {Yifan Yang and Weiquan Huang and Yixuan Wei and Houwen Peng and Xinyang Jiang and Huiqiang Jiang and Fangyun Wei and Yin Wang and Han Hu and Lili Qiu and Yuqing Yang},
  title = {{Attentive Mask CLIP}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2022},
  doi = {10.1109/iccv51070.2023.00260},
}
Figure