Revisiting Self-Supervised Visual Representation Learning

Alexander Kolesnikov, Xiaohua Zhai, Lucas Beyer

Abstract

Unsupervised visual representation learning remains a largely unsolved problem in computer vision research. Among a big body of recently proposed approaches for unsupervised learning of visual representations, a class of self-supervised techniques achieves superior performance on many challenging benchmarks. A large number of the pretext tasks for self-supervised learning have been studied, but other important aspects, such as the choice of convolutional neural networks (CNN), has not received equal attention. Therefore, we revisit numerous previously proposed self-supervised models, conduct a thorough large scale study and, as a result, uncover multiple crucial insights. We challenge a number of common practices in self-supervised visual representation learning and observe that standard recipes for CNN design do not always translate to self-supervised representation learning. As part of our study, we drastically boost the performance of previously proposed techniques and outperform previously published state-of-the-art results by a large margin.

Introduction

In computer vision, self-supervised learning (SSL) aims to learn useful image features from unlabeled data by solving "pretext tasks"—artificial challenges like predicting the rotation of an image or the relative position of patches. Historically, researchers focused almost exclusively on inventing clever new pretext tasks while using a fixed, older architecture (usually AlexNet) for evaluation. This paper argues that this approach is a blind spot. We investigate a fundamental question: How much of our progress in SSL is due to better tasks versus better architectures? We examine whether standard design "best practices" from supervised learning actually hold up when labels are removed.

Methods

We conducted a massive empirical study, revisiting four prominent self-supervised pretext tasks:

  1. Rotation: Predicting if an image was rotated by 0, 90, 180, or 270 degrees.
  2. Exemplar: Learning to be invariant to heavy data augmentations of the same image.
  3. Jigsaw: Recovering the original arrangement of shuffled image patches.
  4. Relative Patch Location: Predicting the spatial relationship between two patches from the same image.

These tasks were evaluated across several modern architectures, including ResNet-50 (v1 and v2), VGG19, and RevNet (an invertible architecture). We specifically varied the widening factor ($k$), which increases the number of filters in each layer, to observe how model capacity affects representation quality. To evaluate the learned features, we used the standard linear probe method: freezing the weights of the self-supervised model and training a simple linear classifier on top using labeled data from ImageNet (ILSVRC-2012) and Places205.

Key Insights

Architecture-Task Interaction

Our study reveals that there is no "one size fits all" architecture for self-supervised learning. The ranking of SSL methods depends heavily on the backbone used. For instance, while ResNet-50 v1 performs best for the Relative Patch Location task, it is suboptimal for the Rotation task, where RevNet50 excels. This suggests that the choice of architecture is as critical as the task itself.

Downstream ImageNet accuracy for different architectures across four pretext tasks.

The Impact of Model Width

Increasing the number of filters (model width) consistently and significantly boosts performance in SSL, often more dramatically than it does in supervised learning. This scaling effect suggests that many previous SSL methods were primarily limited by the capacity of their backbones rather than the tasks themselves.

Impact of model width multiplier and representation size on downstream accuracy.

Skip-connections and Feature Quality

In older models like AlexNet or VGG, feature quality usually peaks in the middle layers and degrades toward the end because the model over-specializes on the pretext task. In ResNets, skip-connections prevent this "feature vanishing," and the very last layer before the classifier consistently provides the best representations.

Pretext Performance as a Proxy

Crucially, doing better at the pretext task (e.g., getting 99% accuracy on predicting rotation) does not guarantee better features for downstream tasks like object recognition. This suggests that pretext task accuracy is a poor metric for evaluating representation quality during the development of new SSL methods.

Key Results

By pairing older pretext tasks with modern, wider architectures, we achieved results that outperformed the then-current state-of-the-art:

  • Rotation + RevNet50 ($4\times$ wider): Achieved 55.4% top-1 accuracy on ImageNet, compared to the 38.7% originally reported using AlexNet.
  • Relative Patch Location + ResNet50 v1 ($2\times$ wider): Achieved 51.4%, a massive jump from the 36.2% previously reported.
  • Closing the Gap: These results effectively halved the performance gap between self-supervised and fully supervised models at the time of publication.

Limitations and Future Work

While the paper shows that architecture matters, it doesn't provide a "label-free" way to choose the best architecture for a specific task; currently, a labeled validation set is still required for model selection. Additionally, while RevNet helped significantly with the Rotation task, it didn't show consistent gains across all other tasks. Future SSL research should stop looking at pretext tasks in isolation and instead treat the task and the architecture as a single, unified system.

Summary updated

Abstract

Unsupervised visual representation learning remains a largely unsolved problem in computer vision research. Among a big body of recently proposed approaches for unsupervised learning of visual representations, a class of self-supervised techniques achieves superior performance on many challenging benchmarks. A large number of the pretext tasks for self-supervised learning have been studied, but other important aspects, such as the choice of convolutional neural networks (CNN), has not received equal attention. Therefore, we revisit numerous previously proposed self-supervised models, conduct a thorough large scale study and, as a result, uncover multiple crucial insights. We challenge a number of common practices in self-supervised visual representation learning and observe that standard recipes for CNN design do not always translate to self-supervised representation learning. As part of our study, we drastically boost the performance of previously proposed techniques and outperform previously published state-of-the-art results by a large margin. We will release the code for reproducing our experiments when the anonymity requirements are lifted.
BibTeX
@inproceedings{kolesnikov2019revisiting,
  author = {Alexander Kolesnikov and Xiaohua Zhai and Lucas Beyer},
  title = {{Revisiting Self-Supervised Visual Representation Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=uCgjP1VM7u},
}
Figure