Introduction
Self-supervised learning (SSL), particularly contrastive methods like MoCo and SimCLR, has enabled large-scale vision models to learn robust representations from unlabeled data. However, a significant performance gap exists between large models (e.g., ResNet50) and smaller architectures (e.g., MobileNetV3) in the SSL paradigm. While a small model might trail a large one by 9% in supervised settings, this gap expands to nearly 40% in SSL.
The common assumption has been that the instance discrimination pretext task is too complex for low-capacity models, leading researchers to rely on computationally expensive Knowledge Distillation (KD) from pre-trained "teacher" models. This paper challenges that assumption, demonstrating that small models fail not because they cannot solve the task, but because they solve it "too well" in a way that hinders semantic generalization.
The "Easy Task" Paradox and Over-clustering
A key finding of this work is the "Easy Task" paradox: small models often achieve higher accuracy on the pretext task (instance discrimination) than their larger counterparts. Despite this, their linear probing performance on ImageNet is significantly worse.
Diagnostic metrics—including alignment, uniformity, and intra-class alignment—reveal a phenomenon termed over-clustering. Small models map augmented versions of the same image to nearly identical points in the feature space (perfect alignment) but fail to group semantically similar images together. This results in a representation space characterized by extremely tight, isolated points rather than meaningful semantic clusters.
Proposed Recipe for Small SSL
To mitigate over-clustering and improve generalization without distillation signals, the authors propose a specific set of architectural and hyperparameter adjustments:
- Lower Temperature ($\tau$): Reducing the temperature (e.g., to 0.05) helps small models converge more effectively by sharpening the contrastive distribution.
- Wider MLP Projectors: Increasing the width of the projection head (e.g., doubling the hidden layer size) prevents an "Information Bottleneck" that might otherwise discard semantically useful features.
- Stronger Augmentation: Enhanced color jitter and blur prevent the model from relying on low-level "shortcuts" like color histograms to solve the instance discrimination task.
- Initialization Warm-up: A brief period of distillation (as few as 2 epochs) can help the model escape poor local minima, though the goal is to minimize reliance on teacher signals.
Experimental Results
The proposed "recipe" yields substantial improvements across a variety of small architectures, including ResNet18, MobileNetV3, EfficientNet-B0, and DeiT-Tiny. On ImageNet linear probing, these modifications boost performance by over 14% on average without any long-term distillation.
| Model | Baseline SSL | Improved SSL | Gain |
|---|---|---|---|
| MobileNetV3-Large | 32.2% | 48.7% | +16.5% |
| MobileNetV3-Small | 26.8% | 41.3% | +14.5% |
| DeiT-Tiny | 23.4% | 38.6% | +15.2% |
| EfficientNet-B0 | 42.1% | 55.9% | +13.8% |
Beyond ImageNet, these improvements generalize to transfer learning tasks on datasets such as CIFAR-10, CIFAR-100, and Caltech-101, significantly outperforming standard SSL baselines.
Conclusion
This research demonstrates that small models are capable of effective self-supervised learning without the need for heavy distillation signals. By identifying and addressing the issue of over-clustering through strategic hyperparameter and architectural choices, the performance gap between large and small SSL models can be significantly narrowed. Future work remains to bridge the remaining gap to distillation-based methods and to further explore the theoretical relationship between model capacity and contrastive temperature.
Summary updated