Self-Supervised Visual Representation Learning via Residual Momentum

Trung X. Pham, Axi Niu, Kang Zhang, Sultan Rizky Hikmawan Madjid, Ji Woo Hong, Daehyeok Kim, Joshua Tian Jin Tee, Chang D. Yoo

TL;DR

This paper introduces "residual momentum" (Res-MoCo) to address a performance gap between the "student" and "teacher" networks in self-supervised learning (SSL). By explicitly minimizing the representation difference between these two networks when processing the same input, Res-MoCo significantly boosts performance on various downstream tasks, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Self-supervised learning (SSL) has become a powerful paradigm for learning visual representations from unlabeled data, especially through momentum-based frameworks like MoCo, BYOL, and DINO. These methods typically use a Siamese network architecture with two branches: an "online" (student) encoder and a "momentum" (teacher) encoder. The teacher's weights are a slowly updated moving average of the student's weights, aiming to provide stable targets.

However, the authors identify a critical, overlooked issue: a "representation gap" between the student and teacher encoders, even when they process the exact same input image. This "intra-representation gap" is distinct from the "inter-representation gap" (which focuses on the difference between representations of different augmented views of an image), and it's hypothesized to be a bottleneck preventing models from reaching their full potential. Existing SSL methods, while effective, don't directly address this intra-gap, leading to a performance disparity between the student and teacher.

Methods

The core contribution is the proposed "residual momentum" (or "intra-momentum"). This technique aims to directly reduce the intra-representation gap. Instead of focusing solely on making representations of different augmented views similar (inter-momentum), Res-MoCo adds a loss term that minimizes the difference between the outputs of the student and teacher encoders when they receive the identical input.

The authors propose to measure this intra-representation gap using a distance function, with cosine similarity being the most effective. The overall loss for Res-MoCo is a combination of the standard inter-momentum loss (from the baseline MoCo-v3) and the new intra-momentum loss:

$L_{Res-MoCo} = L_{Inter-M} + L_{Intra-M}$

This approach is designed to be simple, plug-and-play, and can be integrated into existing SSL frameworks without major architectural changes.

Datasets

The paper evaluates Res-MoCo on several standard datasets:

  • CIFAR-10 (10 classes)
  • CIFAR-100 (100 classes)
  • ImageNet-100 (100 classes)
  • ImageNet-1K (1000 classes)

For transfer learning tasks, they also use:

  • VOC07+12 (object detection)
  • COCO2017 (object detection)

Key Insights and Ideas

  • The "Invisible" Intra-Representation Gap: The most significant insight is the identification and quantification of the representation gap between student and teacher encoders for the same input, which was previously overlooked.
  • Intra-Momentum as a Solution: Directly minimizing this intra-gap with "residual momentum" proves to be a highly effective strategy for improving SSL performance.
  • Augmentation Agnostic: The proposed intra-momentum is augmentation-agnostic, making it flexible and easy to integrate.
  • Teacher-Student Synergy: By forcing the student to closely match the teacher's output, both models are improved, creating a positive feedback loop.
  • Longer Training Benefits: Intra-momentum shows particularly strong benefits in longer training regimes (e.g., 1000 epochs), where the teacher-student gap can become more pronounced.

Key Results

  • Improved Linear Classification: Res-MoCo consistently outperforms the MoCo-v3 baseline across various datasets and network architectures (ResNet-18, ResNet-34, ResNet-50). For example, on CIFAR-10, it achieves 93.71% top-1 accuracy (+0.61% over MoCo-v3), and on ImageNet-100, it achieves 82.62% top-1 accuracy (+1.36% over MoCo-v3).
  • Enhanced Object Detection: When transferring representations to object detection tasks (VOC07+12 and COCO), Res-MoCo shows clear improvements over MoCo-v3, surpassing other competitors in COCO for AP, AP50, and AP75.
  • Reduced Representation Gap: Visualizations (Figures 2, 4, 6, 7, 8, 12) clearly demonstrate that Res-MoCo significantly reduces the intra-representation gap between the teacher and student compared to MoCo-v3.
  • Cleaner Feature Maps: Learned feature maps (Figures 9, 13, 14) show that Res-MoCo produces cleaner representations, focusing more on objects and removing background noise, which is crucial for downstream tasks like object detection.
  • More Accurate Attention Maps: GradCAM visualizations (Figures 10, 15, 16) indicate that Res-MoCo generates more accurate attention heatmaps, highlighting its improved ability to focus on relevant image regions.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper states no additional hyperparameters are introduced, the effectiveness of intra-momentum might still be sensitive to the choice of distance function and the momentum coefficient $\beta$. The paper does explore different distance functions, finding cosine similarity to be optimal.
  • Longer Training Advantage: The paper highlights that intra-momentum particularly benefits longer training. Its impact on very short training schedules might be less pronounced, though still beneficial.
  • Future Work: The authors suggest that the representation gap between teacher and student should be more carefully considered in future SSL research for better representation learning. The plug-and-play nature of intra-momentum suggests it can be applied to a wider range of SSL frameworks, both momentum-based and momentum-free.

Summary updated

Abstract

Self-supervised learning (SSL) approaches have shown promising capabilities in learning the representation from unlabeled data. Amongst them, momentum-based frameworks have attracted significant attention. Despite being a great success, these momentum-based SSL frameworks suffer from a large gap in representation between the online encoder (student) and the momentum encoder (teacher), which hinders performance on downstream tasks. This paper is the first to investigate and identify this invisible gap as a bottleneck that has been overlooked in the existing SSL frameworks, potentially preventing the models from learning good representation. To solve this problem, we propose "residual momentum" to directly reduce this gap to encourage the student to learn the representation as close to that of the teacher as possible, narrow the performance gap with the teacher, and significantly improve the existing SSL. Our method is straightforward, easy to implement, and can be easily plugged into other SSL frameworks. Extensive experimental results on numerous benchmark datasets and diverse network architectures have demonstrated the effectiveness of our method over the state-of-the-art contrastive learning baselines.
BibTeX
@article{pham2023self,
  author = {Trung X. Pham and Axi Niu and Kang Zhang and Sultan Rizky Hikmawan Madjid and Ji Woo Hong and Daehyeok Kim and Joshua Tian Jin Tee and Chang D. Yoo},
  title = {{Self-Supervised Visual Representation Learning via Residual Momentum}},
  journal = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3325842},
}
Figure