Abstract
Contrastive Language-Image Pre-training (CLIP) models demonstrate exceptional zero-shot recognition capabilities but often lag behind Masked Image Modeling (MIM) approaches when fine-tuned for dense downstream tasks such as object detection and semantic segmentation. This paper introduces FD-CLIP, a feature map distillation framework designed to bridge this performance gap. By distilling token-level representations from a frozen CLIP teacher into a student model, FD-CLIP successfully combines the semantic richness of CLIP with the superior fine-tuning stability and localization of MIM. Experimental results on ImageNet-1K, COCO, and ADE20K demonstrate that FD-CLIP consistently outperforms standard CLIP fine-tuning, achieving state-of-the-art results when scaled to large-scale models.
Introduction
In modern vision pre-training, two paradigms dominate: CLIP, which learns high-level semantics from massive image-text pairs, and MIM (e.g., MAE), which learns fine-grained spatial features by reconstructing image patches. While CLIP is a leader in zero-shot tasks, its image-level contrastive objective often results in representations that lack the token-level granularity required for dense prediction tasks. Conversely, MIM excels at localization but lacks semantic labels.
The authors of this paper identify "target granularity" as the primary bottleneck for CLIP's fine-tuning performance. They propose a distillation-based approach to move from image-level to token-level supervision, effectively making CLIP as effective as MIM for downstream fine-tuning without sacrificing its semantic knowledge.
The FD-CLIP Framework
The core of the proposed method is Feature Map Distillation (FD). Instead of training with a contrastive loss, a student model is trained to mimic the entire output feature map of a pre-trained, frozen CLIP teacher.
Key technical components include:
- Token-level Distillation: The student learns from all tokens in the teacher's feature map, forcing the acquisition of localized representations.
- Feature Map Standardization: Normalizing the teacher’s feature maps stabilizes training and amplifies subtle semantic information.
- Asymmetric Drop Path: Stochastic depth regularization is applied only to the student, improving generalization while maintaining a clear signal from the teacher.
- Shared Relative Position Bias (RPB): Replacing absolute position encodings with relative bias enhances the model's translational invariance.
Experimental Setup
Models were distilled on ImageNet-1K (1.28M images) for 300 epochs. The framework was evaluated across several benchmarks:
- Classification: ImageNet-1K (fine-tuning and linear probing).
- Dense Prediction: ADE20K (segmentation), COCO (detection/segmentation), and NYUv2 (depth estimation).
- Scaling: Large-scale training using WIT-400M and ImageNet-22K, scaling up to a SwinV2-G model with 3 billion parameters.
Analysis and Insights
The success of FD-CLIP is attributed to several structural and optimization improvements:
- Granularity: Token-level supervision is the "secret sauce" that grants CLIP the fine-tuning advantages typically associated with MIM.
- Spatial Reasoning: By using relative position bias and distillation, FD-CLIP eliminates the "vertical bar" artifacts common in original CLIP attention maps, which are caused by over-sensitivity to absolute coordinates.
- Attention Diversity: FD-CLIP maintains higher diversity among attention heads in deeper layers, a characteristic shared with MIM models that aids transferability.
- Optimization Stability: Visualization of the loss landscape reveals that FD-CLIP converges to a significantly flatter minimum compared to original CLIP.
Key Results
FD-CLIP demonstrates consistent gains over original CLIP across all evaluated tasks:
- ImageNet-1K: A ViT-B/16 student reached 85.0% top-1 accuracy (+2.1% over CLIP).
- COCO Detection: Achieved a +3.2 point improvement in box AP.
- ADE20K Segmentation: Improved mIoU by +2.2 points.
- State-of-the-Art Scaling: When applied to SwinV2-G, the method achieved a record 64.2 mAP on COCO.
- Semantic Preservation: The model maintained or improved upon CLIP’s zero-shot and linear probing performance.
Conclusion and Future Work
While FD-CLIP significantly improves fine-tuning performance, it introduces a ~3% increase in training cost due to the distillation step and remains dependent on the quality of the pre-trained teacher. Future research may explore integrating this distillation process directly into the initial pre-training phase to streamline the pipeline.
Summary updated