Improving CLIP Fine-tuning Performance

Yixuan Wei, Han Hu, Zhenda Xie, Ze Liu, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, Baining Guo

Abstract

Contrastive Language-Image Pre-training (CLIP) models demonstrate exceptional zero-shot recognition capabilities but often lag behind Masked Image Modeling (MIM) approaches when fine-tuned for dense downstream tasks such as object detection and semantic segmentation. This paper introduces FD-CLIP, a feature map distillation framework designed to bridge this performance gap. By distilling token-level representations from a frozen CLIP teacher into a student model, FD-CLIP successfully combines the semantic richness of CLIP with the superior fine-tuning stability and localization of MIM. Experimental results on ImageNet-1K, COCO, and ADE20K demonstrate that FD-CLIP consistently outperforms standard CLIP fine-tuning, achieving state-of-the-art results when scaled to large-scale models.

Introduction

In modern vision pre-training, two paradigms dominate: CLIP, which learns high-level semantics from massive image-text pairs, and MIM (e.g., MAE), which learns fine-grained spatial features by reconstructing image patches. While CLIP is a leader in zero-shot tasks, its image-level contrastive objective often results in representations that lack the token-level granularity required for dense prediction tasks. Conversely, MIM excels at localization but lacks semantic labels.

The authors of this paper identify "target granularity" as the primary bottleneck for CLIP's fine-tuning performance. They propose a distillation-based approach to move from image-level to token-level supervision, effectively making CLIP as effective as MIM for downstream fine-tuning without sacrificing its semantic knowledge.

The FD-CLIP Framework

The core of the proposed method is Feature Map Distillation (FD). Instead of training with a contrastive loss, a student model is trained to mimic the entire output feature map of a pre-trained, frozen CLIP teacher.

The FD-CLIP distillation framework, illustrating the teacher-student architecture and feature map standardization.

Key technical components include:

  • Token-level Distillation: The student learns from all tokens in the teacher's feature map, forcing the acquisition of localized representations.
  • Feature Map Standardization: Normalizing the teacher’s feature maps stabilizes training and amplifies subtle semantic information.
  • Asymmetric Drop Path: Stochastic depth regularization is applied only to the student, improving generalization while maintaining a clear signal from the teacher.
  • Shared Relative Position Bias (RPB): Replacing absolute position encodings with relative bias enhances the model's translational invariance.

Experimental Setup

Models were distilled on ImageNet-1K (1.28M images) for 300 epochs. The framework was evaluated across several benchmarks:

  • Classification: ImageNet-1K (fine-tuning and linear probing).
  • Dense Prediction: ADE20K (segmentation), COCO (detection/segmentation), and NYUv2 (depth estimation).
  • Scaling: Large-scale training using WIT-400M and ImageNet-22K, scaling up to a SwinV2-G model with 3 billion parameters.

Analysis and Insights

The success of FD-CLIP is attributed to several structural and optimization improvements:

  1. Granularity: Token-level supervision is the "secret sauce" that grants CLIP the fine-tuning advantages typically associated with MIM.
  2. Spatial Reasoning: By using relative position bias and distillation, FD-CLIP eliminates the "vertical bar" artifacts common in original CLIP attention maps, which are caused by over-sensitivity to absolute coordinates.
  3. Attention Diversity: FD-CLIP maintains higher diversity among attention heads in deeper layers, a characteristic shared with MIM models that aids transferability.
  4. Optimization Stability: Visualization of the loss landscape reveals that FD-CLIP converges to a significantly flatter minimum compared to original CLIP.
Loss landscape and accuracy curves comparing MAE, CLIP, and FD-CLIP, showing the flatter minimum achieved by FD-CLIP.

Key Results

FD-CLIP demonstrates consistent gains over original CLIP across all evaluated tasks:

  • ImageNet-1K: A ViT-B/16 student reached 85.0% top-1 accuracy (+2.1% over CLIP).
  • COCO Detection: Achieved a +3.2 point improvement in box AP.
  • ADE20K Segmentation: Improved mIoU by +2.2 points.
  • State-of-the-Art Scaling: When applied to SwinV2-G, the method achieved a record 64.2 mAP on COCO.
  • Semantic Preservation: The model maintained or improved upon CLIP’s zero-shot and linear probing performance.

Conclusion and Future Work

While FD-CLIP significantly improves fine-tuning performance, it introduces a ~3% increase in training cost due to the distillation step and remains dependent on the quality of the pre-trained teacher. Future research may explore integrating this distillation process directly into the initial pre-training phase to streamline the pipeline.

Summary updated

Abstract

CLIP models have demonstrated impressively high zero-shot recognition accuracy, however, their fine-tuning performance on downstream vision tasks is sub-optimal. Contrarily, masked image modeling (MIM) performs exceptionally for fine-tuning on downstream tasks, despite the absence of semantic labels during training. We note that the two tasks have different ingredients: image-level targets versus token-level targets, a cross-entropy loss versus a regression loss, and full-image inputs versus partial-image inputs. To mitigate the differences, we introduce a classical feature map distillation framework, which can simultaneously inherit the semantic capability of CLIP models while constructing a task incorporated key ingredients of MIM. Experiments suggest that the feature map distillation approach significantly boosts the fine-tuning performance of CLIP models on several typical down-stream vision tasks. We also observe that the approach yields new CLIP representations which share some diagnostic properties with those of MIM. Furthermore, the feature map distillation approach generalizes to other pre-training models, such as DINO, DeiT and SwinV2-G, reaching a new record of 64.2 mAP on COCO object detection with +1.1 improvement. The code and models are publicly available at https://github.com/SwinTransformer/Feature-Distillation.
BibTeX
@inproceedings{wei2023improving,
  author = {Yixuan Wei and Han Hu and Zhenda Xie and Ze Liu and Zheng Zhang and Yue Cao and Jianmin Bao and Dong Chen and Baining Guo},
  title = {{Improving CLIP Fine-tuning Performance}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00501},
}
Figure