Abstract
iCLIP is a unified pre-training framework that bridges the gap between traditional image classification (IC) and contrastive language-image pre-training (CLIP). By sharing a single text encoder across both tasks and enriching short category labels with detailed dictionary definitions, iCLIP achieves the sharp discriminative power of supervised learning alongside the flexible zero-shot capabilities of CLIP. Experimental results demonstrate state-of-the-art performance on ImageNet-1K and significant gains in zero-shot classification and retrieval.
Introduction
In computer vision, two dominant paradigms exist: Image Classification (IC) and Contrastive Language-Image Pre-training (CLIP). IC relies on clean, human-annotated datasets like ImageNet, producing highly discriminative features but remaining limited to fixed categories. Conversely, CLIP learns from noisy web-crawled image-text pairs, offering excellent zero-shot capabilities but often lacking precise conceptual boundaries. Previous attempts at "naive multi-tasking" using separate heads for these tasks yielded marginal gains. iCLIP addresses this by deeply fusing these paradigms into a single, unified formula.
Methodology
The iCLIP framework introduces three primary adaptations to unify IC and CLIP:
- Deep Fusion via Shared Encoders: Unlike standard classification which uses a linear layer, iCLIP employs a shared Text Encoder (e.g., RoBERTa) for both class labels and captions. This forces the model to use identical logic for understanding "labels" and "captions."
- Unified Loss Function: By converting the classification task to use cosine similarity instead of inner-product, iCLIP optimizes both tasks using a single contrastive loss function (InfoNCE).
- Dictionary Enhancement: To bridge the granularity gap between short labels (e.g., "night bird") and descriptive captions, iCLIP enriches labels using WordNet definitions. For instance, "night bird" becomes "A photo of a night bird, any bird associated with night, like owl, nightingale, nighthawk."
To manage the computational overhead of processing thousands of categories, a distributed strategy was implemented to spread text embeddings across multiple GPUs.
Experimental Results
iCLIP was evaluated across diverse benchmarks, including ImageNet-1K, zero-shot classification (14-dataset average), and retrieval tasks (Flickr30K, MSCOCO).
- Classification Performance: iCLIP reached 82.9% top-1 accuracy on ImageNet-1K.
- Zero-Shot Superiority: On the Kornblith 12-dataset benchmark, iCLIP outperformed the original CLIP by 1.8% while maintaining a similar model size.
- Retrieval Gains: Dictionary enhancement led to a 10.7% improvement in top-1 recall on Flickr30K compared to versions without it.
- Robustness: The model showed significant improvements on out-of-distribution datasets like ImageNet-Sketch, proving resilience to visual style changes.
Conclusion
iCLIP demonstrates that human-labeled "gold" data and web-crawled "rich" captions are complementary. By unifying these through shared encoders and knowledge injection via dictionary definitions, iCLIP provides a robust foundation for visual recognition that excels in both supervised and zero-shot settings. Future work will explore scaling this approach to billion-parameter foundation models and further automating knowledge injection.
Summary updated