TL;DR
This paper introduces "residual momentum" (Res-MoCo) to address a performance gap between the "student" and "teacher" networks in self-supervised learning (SSL). By explicitly minimizing the representation difference between these two networks when processing the same input, Res-MoCo significantly boosts performance on various downstream tasks, outperforming existing state-of-the-art methods.
What Problem Does This Paper Address?
Self-supervised learning (SSL) has become a powerful paradigm for learning visual representations from unlabeled data, especially through momentum-based frameworks like MoCo, BYOL, and DINO. These methods typically use a Siamese network architecture with two branches: an "online" (student) encoder and a "momentum" (teacher) encoder. The teacher's weights are a slowly updated moving average of the student's weights, aiming to provide stable targets.
However, the authors identify a critical, overlooked issue: a "representation gap" between the student and teacher encoders, even when they process the exact same input image. This "intra-representation gap" is distinct from the "inter-representation gap" (which focuses on the difference between representations of different augmented views of an image), and it's hypothesized to be a bottleneck preventing models from reaching their full potential. Existing SSL methods, while effective, don't directly address this intra-gap, leading to a performance disparity between the student and teacher.
Methods
The core contribution is the proposed "residual momentum" (or "intra-momentum"). This technique aims to directly reduce the intra-representation gap. Instead of focusing solely on making representations of different augmented views similar (inter-momentum), Res-MoCo adds a loss term that minimizes the difference between the outputs of the student and teacher encoders when they receive the identical input.
The authors propose to measure this intra-representation gap using a distance function, with cosine similarity being the most effective. The overall loss for Res-MoCo is a combination of the standard inter-momentum loss (from the baseline MoCo-v3) and the new intra-momentum loss:
$L_{Res-MoCo} = L_{Inter-M} + L_{Intra-M}$
This approach is designed to be simple, plug-and-play, and can be integrated into existing SSL frameworks without major architectural changes.
Datasets
The paper evaluates Res-MoCo on several standard datasets:
- CIFAR-10 (10 classes)
- CIFAR-100 (100 classes)
- ImageNet-100 (100 classes)
- ImageNet-1K (1000 classes)
For transfer learning tasks, they also use:
- VOC07+12 (object detection)
- COCO2017 (object detection)
Key Insights and Ideas
- The "Invisible" Intra-Representation Gap: The most significant insight is the identification and quantification of the representation gap between student and teacher encoders for the same input, which was previously overlooked.
- Intra-Momentum as a Solution: Directly minimizing this intra-gap with "residual momentum" proves to be a highly effective strategy for improving SSL performance.
- Augmentation Agnostic: The proposed intra-momentum is augmentation-agnostic, making it flexible and easy to integrate.
- Teacher-Student Synergy: By forcing the student to closely match the teacher's output, both models are improved, creating a positive feedback loop.
- Longer Training Benefits: Intra-momentum shows particularly strong benefits in longer training regimes (e.g., 1000 epochs), where the teacher-student gap can become more pronounced.
Key Results
- Improved Linear Classification: Res-MoCo consistently outperforms the MoCo-v3 baseline across various datasets and network architectures (ResNet-18, ResNet-34, ResNet-50). For example, on CIFAR-10, it achieves 93.71% top-1 accuracy (+0.61% over MoCo-v3), and on ImageNet-100, it achieves 82.62% top-1 accuracy (+1.36% over MoCo-v3).
- Enhanced Object Detection: When transferring representations to object detection tasks (VOC07+12 and COCO), Res-MoCo shows clear improvements over MoCo-v3, surpassing other competitors in COCO for AP, AP50, and AP75.
- Reduced Representation Gap: Visualizations (Figures 2, 4, 6, 7, 8, 12) clearly demonstrate that Res-MoCo significantly reduces the intra-representation gap between the teacher and student compared to MoCo-v3.
- Cleaner Feature Maps: Learned feature maps (Figures 9, 13, 14) show that Res-MoCo produces cleaner representations, focusing more on objects and removing background noise, which is crucial for downstream tasks like object detection.
- More Accurate Attention Maps: GradCAM visualizations (Figures 10, 15, 16) indicate that Res-MoCo generates more accurate attention heatmaps, highlighting its improved ability to focus on relevant image regions.
Limitations and Future Work
- Hyperparameter Sensitivity: While the paper states no additional hyperparameters are introduced, the effectiveness of intra-momentum might still be sensitive to the choice of distance function and the momentum coefficient $\beta$. The paper does explore different distance functions, finding cosine similarity to be optimal.
- Longer Training Advantage: The paper highlights that intra-momentum particularly benefits longer training. Its impact on very short training schedules might be less pronounced, though still beneficial.
- Future Work: The authors suggest that the representation gap between teacher and student should be more carefully considered in future SSL research for better representation learning. The plug-and-play nature of intra-momentum suggests it can be applied to a wider range of SSL frameworks, both momentum-based and momentum-free.
Summary updated