iCLIP: Bridging Image Classification and Contrastive Language-Image Pre-training for Visual Recognition

Yixuan Wei, Yue Cao, Zheng Zhang, Houwen Peng, Zhuliang Yao, Zhenda Xie, Han Hu, Baining Guo

Abstract

iCLIP is a unified pre-training framework that bridges the gap between traditional image classification (IC) and contrastive language-image pre-training (CLIP). By sharing a single text encoder across both tasks and enriching short category labels with detailed dictionary definitions, iCLIP achieves the sharp discriminative power of supervised learning alongside the flexible zero-shot capabilities of CLIP. Experimental results demonstrate state-of-the-art performance on ImageNet-1K and significant gains in zero-shot classification and retrieval.

Introduction

In computer vision, two dominant paradigms exist: Image Classification (IC) and Contrastive Language-Image Pre-training (CLIP). IC relies on clean, human-annotated datasets like ImageNet, producing highly discriminative features but remaining limited to fixed categories. Conversely, CLIP learns from noisy web-crawled image-text pairs, offering excellent zero-shot capabilities but often lacking precise conceptual boundaries. Previous attempts at "naive multi-tasking" using separate heads for these tasks yielded marginal gains. iCLIP addresses this by deeply fusing these paradigms into a single, unified formula.

Methodology

The iCLIP framework introduces three primary adaptations to unify IC and CLIP:

  1. Deep Fusion via Shared Encoders: Unlike standard classification which uses a linear layer, iCLIP employs a shared Text Encoder (e.g., RoBERTa) for both class labels and captions. This forces the model to use identical logic for understanding "labels" and "captions."
  2. Unified Loss Function: By converting the classification task to use cosine similarity instead of inner-product, iCLIP optimizes both tasks using a single contrastive loss function (InfoNCE).
  3. Dictionary Enhancement: To bridge the granularity gap between short labels (e.g., "night bird") and descriptive captions, iCLIP enriches labels using WordNet definitions. For instance, "night bird" becomes "A photo of a night bird, any bird associated with night, like owl, nightingale, nighthawk."
Overview of the iCLIP framework, highlighting the shared encoders and dictionary-enhanced class labels.

To manage the computational overhead of processing thousands of categories, a distributed strategy was implemented to spread text embeddings across multiple GPUs.

Experimental Results

iCLIP was evaluated across diverse benchmarks, including ImageNet-1K, zero-shot classification (14-dataset average), and retrieval tasks (Flickr30K, MSCOCO).

  • Classification Performance: iCLIP reached 82.9% top-1 accuracy on ImageNet-1K.
  • Zero-Shot Superiority: On the Kornblith 12-dataset benchmark, iCLIP outperformed the original CLIP by 1.8% while maintaining a similar model size.
  • Retrieval Gains: Dictionary enhancement led to a 10.7% improvement in top-1 recall on Flickr30K compared to versions without it.
  • Robustness: The model showed significant improvements on out-of-distribution datasets like ImageNet-Sketch, proving resilience to visual style changes.
Zero-shot classification and retrieval performance of iCLIP compared to standard CLIP across various training data configurations.

Conclusion

iCLIP demonstrates that human-labeled "gold" data and web-crawled "rich" captions are complementary. By unifying these through shared encoders and knowledge injection via dictionary definitions, iCLIP provides a robust foundation for visual recognition that excels in both supervised and zero-shot settings. Future work will explore scaling this approach to billion-parameter foundation models and further automating knowledge injection.

Summary updated

Abstract

This paper presents a method that effectively combines two prevalent visual recognition methods, i.e., image classification and contrastive language-image pre-training, dubbed iCLIP. Instead of naïve multi-task learning that use two separate heads for each task, we fuse the two tasks in a deep fashion that adapts the image classification to share the same formula and the same model weights with the language-image pre-training. To further bridge these two tasks, we propose to enhance the category names in image classification tasks using external knowledge, such as their descriptions in dictionaries. Extensive experiments show that the proposed method combines the advantages of two tasks well: the strong discrimination ability in image classification tasks due to the clean category labels, and the good zero-shot ability in CLIP tasks ascribed to the richer semantics in the text descriptions. In particular, it reaches 82.9% top-1 accuracy on IN-1K, and mean-while surpasses CLIP by 1.8%, with similar model size, on zero-shot recognition of Kornblith 12-dataset benchmark. The code and models are publicly available at https://github.com/weiyx16/iCLIP.
BibTeX
@inproceedings{wei2023iclip,
  author = {Yixuan Wei and Yue Cao and Zheng Zhang and Houwen Peng and Zhuliang Yao and Zhenda Xie and Han Hu and Baining Guo},
  title = {{iCLIP: Bridging Image Classification and Contrastive Language-Image Pre-training for Visual Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00272},
}
Figure