Introduction
Self-supervised learning (SSL) has revolutionized computer vision, yet most methods treat Vision Transformers (ViTs) as black boxes, focusing solely on the final layer's output. This approach overlooks a fundamental property of ViTs: their global attention mechanism allows even shallow layers to capture long-range dependencies. Multi-level Contrastive Learning for Vision Transformers (MCVT) addresses this by introducing a hierarchical supervision strategy that leverages intermediate features to learn more robust representations.
Methodology
MCVT departs from traditional single-loss SSL by attaching auxiliary projection heads to multiple stages of the transformer backbone. The core innovation is the MCVT-mix strategy, which applies different types of contrastive losses based on the depth of the layer:
- Low-level Supervision (Stages 1 to $s-1$): Early layers utilize InfoNCE, an instance-based contrastive loss. This forces the model to recognize invariance across different augmented views of the same image, effectively capturing fine-grained, low-level details like textures and local shapes.
- High-level Supervision (Final Stage $s$): The deepest layer employs ProtoNCE (Prototypical Contrastive Learning). By clustering features into semantic prototypes, this loss encourages the model to capture broader category-level relationships, mapping related images (e.g., different breeds of dogs) to similar regions in the embedding space.
The framework is compatible with various ViT architectures, including standard ViT-S and Swin-Transformer (Swin-T).
Experimental Results
MCVT-mix demonstrates superior performance and efficiency across multiple benchmarks:
- Image Classification: On ImageNet-100, MCVT-mix achieved 91.26% Top-1 accuracy with a Swin-T backbone, significantly outperforming the MoBY baseline (87.92%). On ImageNet-1K, it reached 71.6% Top-1 accuracy in just 100 epochs, matching the performance of other models that require much longer training cycles.
- Object Detection and Segmentation: When evaluated on MS-COCO using Cascade Mask R-CNN, MCVT-mix achieved a Box AP of 48.6, surpassing the fully supervised baseline (48.1). Similarly, on ADE20K, it reached 45.76 mIoU, nearly matching supervised performance.
Key Insights and Ideas
The success of MCVT stems from two primary factors. First, it exploits the "global-at-any-depth" nature of ViTs, providing explicit guidance to early global features that CNN-based SSL methods typically ignore. Second, the hierarchical "mix" of losses aligns with the natural progression of feature abstraction in deep networks—instance-level details in early layers and semantic prototypes in deeper layers.
Analysis using Centered Kernel Alignment (CKA) reveals that MCVT-trained models develop internal representations that are remarkably similar to those of fully supervised models. This suggests that multi-level contrastive signals help the model learn a feature hierarchy that closely mirrors human-annotated semantic structures.
Limitations and Future Work
While effective, MCVT introduces additional hyperparameters, such as the number of clusters ($K$) in ProtoNCE, which may require tuning for different datasets. Additionally, the computational cost of K-means clustering and multiple projection heads is a factor, though it is largely mitigated by the model's faster convergence. Future research could explore how these multi-level signals scale to massive, "web-scale" datasets where the trade-offs between local and global features might evolve.
Summary updated