Multi-level Contrastive Learning for Self-Supervised Vision Transformers

Shentong Mo, Zhun Sun, Chao Li

Introduction

Self-supervised learning (SSL) has revolutionized computer vision, yet most methods treat Vision Transformers (ViTs) as black boxes, focusing solely on the final layer's output. This approach overlooks a fundamental property of ViTs: their global attention mechanism allows even shallow layers to capture long-range dependencies. Multi-level Contrastive Learning for Vision Transformers (MCVT) addresses this by introducing a hierarchical supervision strategy that leverages intermediate features to learn more robust representations.

Methodology

MCVT departs from traditional single-loss SSL by attaching auxiliary projection heads to multiple stages of the transformer backbone. The core innovation is the MCVT-mix strategy, which applies different types of contrastive losses based on the depth of the layer:

  1. Low-level Supervision (Stages 1 to $s-1$): Early layers utilize InfoNCE, an instance-based contrastive loss. This forces the model to recognize invariance across different augmented views of the same image, effectively capturing fine-grained, low-level details like textures and local shapes.
  2. High-level Supervision (Final Stage $s$): The deepest layer employs ProtoNCE (Prototypical Contrastive Learning). By clustering features into semantic prototypes, this loss encourages the model to capture broader category-level relationships, mapping related images (e.g., different breeds of dogs) to similar regions in the embedding space.
The MCVT architecture featuring multi-stage supervision with low-level and high-level projection heads.

The framework is compatible with various ViT architectures, including standard ViT-S and Swin-Transformer (Swin-T).

Experimental Results

MCVT-mix demonstrates superior performance and efficiency across multiple benchmarks:

  • Image Classification: On ImageNet-100, MCVT-mix achieved 91.26% Top-1 accuracy with a Swin-T backbone, significantly outperforming the MoBY baseline (87.92%). On ImageNet-1K, it reached 71.6% Top-1 accuracy in just 100 epochs, matching the performance of other models that require much longer training cycles.
  • Object Detection and Segmentation: When evaluated on MS-COCO using Cascade Mask R-CNN, MCVT-mix achieved a Box AP of 48.6, surpassing the fully supervised baseline (48.1). Similarly, on ADE20K, it reached 45.76 mIoU, nearly matching supervised performance.

Key Insights and Ideas

The success of MCVT stems from two primary factors. First, it exploits the "global-at-any-depth" nature of ViTs, providing explicit guidance to early global features that CNN-based SSL methods typically ignore. Second, the hierarchical "mix" of losses aligns with the natural progression of feature abstraction in deep networks—instance-level details in early layers and semantic prototypes in deeper layers.

Centered Kernel Alignment (CKA) comparison between Supervised, MOBY, and MCVT models.

Analysis using Centered Kernel Alignment (CKA) reveals that MCVT-trained models develop internal representations that are remarkably similar to those of fully supervised models. This suggests that multi-level contrastive signals help the model learn a feature hierarchy that closely mirrors human-annotated semantic structures.

Limitations and Future Work

While effective, MCVT introduces additional hyperparameters, such as the number of clusters ($K$) in ProtoNCE, which may require tuning for different datasets. Additionally, the computational cost of K-means clustering and multiple projection heads is a factor, though it is largely mitigated by the model's faster convergence. Future research could explore how these multi-level signals scale to massive, "web-scale" datasets where the trade-offs between local and global features might evolve.

Summary updated

Abstract

Recent studies aim to establish contrastive self-supervised learning (CSL) algorithms specialized for the family of Vision Transformers (ViTs) to make them function normally as ordinary convolutional-based backbones in the training progress. Despite obtaining promising performance on related downstream tasks, one compelling property of the ViTs is ignored in those approaches. As previous studies have demonstrated, vision transformers benefit from the early stage global attention mechanics, obtaining feature representations that contain information from distant patches, even in their shallow layers. Motivated by this, we present a simple yet effective framework to facilitate the self-supervised feature learning of transformer based vision architectures, namely, Multi-level Contrastive learning for Vision Transformers (MCVT). Specifically, we equip the vision transformers with individual-based (InfoNCE) and prototypical-based (ProtoNCE) contrastive loss in different stages of the architecture to capture low-level invariance and high-level invariance between views of samples, respectively. We conduct extensive experiments to demonstrate the effectiveness of the proposed method, using two well-known vision transformer backbones, on several vision downstream tasks, including linear classification, detection, and semantic segmentation.
BibTeX
@inproceedings{mo2023multi,
  author = {Shentong Mo and Zhun Sun and Chao Li},
  title = {{Multi-level Contrastive Learning for Self-Supervised Vision Transformers}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00280},
}
Figure