Abstract
Unsupervised visual representation learning remains a largely unsolved problem in computer vision research. Among a big body of recently proposed approaches for unsupervised learning of visual representations, a class of self-supervised techniques achieves superior performance on many challenging benchmarks. A large number of the pretext tasks for self-supervised learning have been studied, but other important aspects, such as the choice of convolutional neural networks (CNN), has not received equal attention. Therefore, we revisit numerous previously proposed self-supervised models, conduct a thorough large scale study and, as a result, uncover multiple crucial insights. We challenge a number of common practices in self-supervised visual representation learning and observe that standard recipes for CNN design do not always translate to self-supervised representation learning. As part of our study, we drastically boost the performance of previously proposed techniques and outperform previously published state-of-the-art results by a large margin.
Introduction
Self-supervised learning (SSL) aims to learn useful visual representations from unlabeled data by training models to solve "pretext tasks"—automatically generated puzzles such as predicting image rotation or unscrambling patches. While recent research has focused heavily on designing novel pretext tasks, the impact of the underlying Convolutional Neural Network (CNN) architecture has often been overlooked. Many studies continue to use older backbones like AlexNet, potentially masking the true potential of these methods. This paper presents a large-scale study revisiting popular SSL techniques using modern architectures to understand how depth, width, and connectivity influence representation quality.
Methodology
The study evaluates four prominent pretext tasks:
- Rotation: Predicting image orientation (0°, 90°, 180°, 270°).
- Exemplar: Learning invariant representations via triplet loss on image augmentations.
- Jigsaw: Solving a $3 \times 3$ shuffled grid of image patches.
- Relative Patch Location: Predicting the spatial relationship between two patches from the same image.
The researchers tested several "backbone" architectures, including ResNet50 (v1 and v2), VGG19, and RevNet50. They also explored the effect of model width by applying a widening factor ($k$) to the number of filters. Performance was measured by freezing the SSL-trained weights and training a linear classifier for ImageNet and Places205 classification. Notably, the authors found that standard linear evaluation requires extensive training (up to 500 epochs) to reach true convergence.
The Critical Role of Architecture
A central finding of this work is that the choice of architecture is as significant as the choice of the pretext task itself. There is no "one-size-fits-all" backbone; for instance, an architecture that performs best on the Rotation task may be sub-optimal for Jigsaw or Relative Patch Location. This suggests that pretext tasks should not be studied in isolation from the models that run them.
Skip-Connections and Feature Decay
In traditional architectures like VGG, the quality of features for downstream tasks often peaks in the middle layers and degrades toward the end of the network. This "feature decay" occurs because the final layers become overly specialized to the specific pretext puzzle. However, in ResNet-style architectures, skip-connections appear to act as a memory mechanism, preserving general semantic information throughout the network. This allows the final layers to maintain high-quality representations suitable for downstream classification.
Scaling and Evaluation Insights
Beyond connectivity, the study highlights two other major factors:
- Width as a Scaling Law: Increasing model width (the number of channels) consistently yields significant performance gains. This benefit is even more pronounced in self-supervised learning than in supervised learning, suggesting that SSL methods are particularly hungry for model capacity.
- Pretext vs. Downstream Performance: Accuracy on the pretext task is not a reliable predictor of downstream success across different architectures. While pretext performance might serve as a proxy within a single architecture, it fails as a universal metric for comparing different model designs.
Results and Conclusion
By modernizing the architectural choices for existing SSL methods, the authors achieved a then-state-of-the-art 55.4% top-1 accuracy on ImageNet using a wide RevNet50 and the Rotation task. This result demonstrated that the gap between unsupervised and supervised learning could be halved simply by scaling the model and using appropriate backbones.
The study concludes that future SSL research must consider the task and architecture as a joint system. While scaling up models provides clear benefits, it also introduces significant computational costs and complicates model selection, as pretext performance alone cannot guide the choice of the best representation.
Summary updated