Abstract
Contrastive learning models like CLIP and DINO excel in zero-shot tasks but have historically lagged behind Masked Image Modeling (MIM) approaches in downstream fine-tuning performance. This paper introduces Feature Distillation (FD), a simple yet powerful post-processing step that allows contrastive models to match or exceed MIM performance. By distilling internal feature maps into a student model with specific architectural priors, we achieve state-of-the-art results, including 89.0% top-1 accuracy on ImageNet-1K with a CLIP ViT-L model and record-breaking performance on ADE20K and COCO.
Introduction
In self-supervised computer vision, two paradigms dominate: Masked Image Modeling (MIM) and Contrastive Learning. While contrastive methods provide robust global representations suitable for linear evaluation, they often struggle during full-parameter fine-tuning for dense tasks like object detection or segmentation. This research investigates the root causes of this performance gap and proposes a distillation-based remedy to enhance the "optimization friendliness" of contrastive representations.
Methodology: Feature Distillation (FD)
The proposed Feature Distillation (FD) framework treats a pre-trained contrastive model as a "teacher" to train a "student" model from scratch. Unlike traditional distillation that relies on final classification logits, FD focuses on internal feature maps, making it universally applicable to models without classification heads.
Key technical components include:
- Feature Whitening: Normalizing teacher features to handle varying magnitudes across different pre-training methods.
- Shared Relative Position Bias (RPB): Implementing a shared RPB across all layers to encourage diverse attention patterns.
- Asymmetric Regularization: Applying drop path only to the student branch to maintain a stable training signal from the teacher.
Why It Works: Optimization Friendliness
The authors identify that FD improves the model's internal dynamics in three critical ways:
- Diversifying Attention: In standard contrastive models, attention heads often collapse in deeper layers, focusing on similar regions. FD forces these heads to maintain diverse "attention distances," effectively increasing the model's functional capacity.
- Diagonal Attention Patterns: FD encourages the model to focus on relative positions rather than absolute pixel locations. This is evidenced by stronger diagonal patterns in attention maps, which correlate with better translational invariance.
- Flatter Loss Landscapes: Visualization of the optimization surface reveals that FD leads to a significantly flatter loss landscape, which is a hallmark of better generalization and easier optimization.
Experimental Results
The effectiveness of FD was validated across multiple industry-standard benchmarks:
- ImageNet-1K: A CLIP ViT-L model enhanced with FD reached 89.0% top-1 accuracy, a substantial improvement over standard fine-tuning.
- Dense Prediction Tasks: Using a SwinV2-G (3B parameters) backbone with FD, the authors set new state-of-the-art records on ADE20K (61.4 mIoU) for semantic segmentation and COCO (64.2 mAP) for object detection.
- Closing the Gap: Models like DINO and EsViT, which previously trailed MIM in fine-tuning, became competitive or superior after the FD post-processing phase.
Conclusion
Feature Distillation demonstrates that the perceived inferiority of contrastive learning in fine-tuning is not a fundamental flaw of the representations themselves, but rather an optimization challenge. By decoupling representation learning from optimization friendliness, researchers can continue to scale contrastive models while using FD as a lightweight "upgrade" to ensure peak performance on downstream tasks. While FD introduces a small additional training cost, the performance gains on large-scale benchmarks justify its use as a standard post-processing step for high-performance vision models.
Summary updated