Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation

Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, Baining GuoarXiv 2022 · DOI · OpenReview

Abstract

Contrastive learning models like CLIP and DINO excel in zero-shot tasks but have historically lagged behind Masked Image Modeling (MIM) approaches in downstream fine-tuning performance. This paper introduces Feature Distillation (FD), a simple yet powerful post-processing step that allows contrastive models to match or exceed MIM performance. By distilling internal feature maps into a student model with specific architectural priors, we achieve state-of-the-art results, including 89.0% top-1 accuracy on ImageNet-1K with a CLIP ViT-L model and record-breaking performance on ADE20K and COCO.

Introduction

In self-supervised computer vision, two paradigms dominate: Masked Image Modeling (MIM) and Contrastive Learning. While contrastive methods provide robust global representations suitable for linear evaluation, they often struggle during full-parameter fine-tuning for dense tasks like object detection or segmentation. This research investigates the root causes of this performance gap and proposes a distillation-based remedy to enhance the "optimization friendliness" of contrastive representations.

Methodology: Feature Distillation (FD)

The proposed Feature Distillation (FD) framework treats a pre-trained contrastive model as a "teacher" to train a "student" model from scratch. Unlike traditional distillation that relies on final classification logits, FD focuses on internal feature maps, making it universally applicable to models without classification heads.

The Feature Distillation framework (left) and its impact on attention distance (right).

Key technical components include:

  • Feature Whitening: Normalizing teacher features to handle varying magnitudes across different pre-training methods.
  • Shared Relative Position Bias (RPB): Implementing a shared RPB across all layers to encourage diverse attention patterns.
  • Asymmetric Regularization: Applying drop path only to the student branch to maintain a stable training signal from the teacher.

Why It Works: Optimization Friendliness

The authors identify that FD improves the model's internal dynamics in three critical ways:

  1. Diversifying Attention: In standard contrastive models, attention heads often collapse in deeper layers, focusing on similar regions. FD forces these heads to maintain diverse "attention distances," effectively increasing the model's functional capacity.
  2. Diagonal Attention Patterns: FD encourages the model to focus on relative positions rather than absolute pixel locations. This is evidenced by stronger diagonal patterns in attention maps, which correlate with better translational invariance.
  3. Flatter Loss Landscapes: Visualization of the optimization surface reveals that FD leads to a significantly flatter loss landscape, which is a hallmark of better generalization and easier optimization.
Attention maps before and after feature distillation, highlighting the emergence of structured diagonal patterns.

Experimental Results

The effectiveness of FD was validated across multiple industry-standard benchmarks:

  • ImageNet-1K: A CLIP ViT-L model enhanced with FD reached 89.0% top-1 accuracy, a substantial improvement over standard fine-tuning.
  • Dense Prediction Tasks: Using a SwinV2-G (3B parameters) backbone with FD, the authors set new state-of-the-art records on ADE20K (61.4 mIoU) for semantic segmentation and COCO (64.2 mAP) for object detection.
  • Closing the Gap: Models like DINO and EsViT, which previously trailed MIM in fine-tuning, became competitive or superior after the FD post-processing phase.

Conclusion

Feature Distillation demonstrates that the perceived inferiority of contrastive learning in fine-tuning is not a fundamental flaw of the representations themselves, but rather an optimization challenge. By decoupling representation learning from optimization friendliness, researchers can continue to scale contrastive models while using FD as a lightweight "upgrade" to ensure peak performance on downstream tasks. While FD introduces a small additional training cost, the performance gains on large-scale benchmarks justify its use as a standard post-processing step for high-performance vision models.

Summary updated

Abstract

Masked image modeling (MIM) learns representations with remarkably good fine-tuning performances, overshadowing previous prevalent pre-training approaches such as image classification, instance contrastive learning, and image-text alignment. In this paper, we show that the inferior fine-tuning performance of these pre-training approaches can be significantly improved by a simple post-processing in the form of feature distillation (FD). The feature distillation converts the old representations to new representations that have a few desirable properties just like those representations produced by MIM. These properties, which we aggregately refer to as optimization friendliness, are identified and analyzed by a set of attention- and optimization-related diagnosis tools. With these properties, the new representations show strong fine-tuning performance. Specifically, the contrastive self-supervised learning methods are made as competitive in fine-tuning as the state-of-the-art masked image modeling (MIM) algorithms. The CLIP models' fine-tuning performance is also significantly improved, with a CLIP ViT-L model reaching 89.0% top-1 accuracy on ImageNet-1K classification. On the 3-billion-parameter SwinV2-G model, the fine-tuning accuracy is improved by +1.5 mIoU / +1.1 mAP to 61.4 mIoU / 64.2 mAP on ADE20K semantic segmentation and COCO object detection, respectively, creating new records on both benchmarks. More importantly, our work provides a way for the future research to focus more effort on the generality and scalability of the learnt representations without being pre-occupied with optimization friendliness since it can be enhanced rather easily. The code will be available at https://github.com/SwinTransformer/Feature-Distillation.
BibTeX
@misc{wei2022contrastive,
  author = {Yixuan Wei and Han Hu and Zhenda Xie and Zheng Zhang and Yue Cao and Jianmin Bao and Dong Chen and Baining Guo},
  title = {{Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation}},
  year = {2022},
  eprint = {2205.14141},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.14141},
}
Figure