On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals

Haizhou Shi, Youcai Zhang, Siliang Tang, Wenjie Zhu, Yaqian Li, Yandong Guo, Yueting Zhuang

Introduction

Self-supervised learning (SSL), particularly contrastive methods like MoCo and SimCLR, has enabled large-scale vision models to learn robust representations from unlabeled data. However, a significant performance gap exists between large models (e.g., ResNet50) and smaller architectures (e.g., MobileNetV3) in the SSL paradigm. While a small model might trail a large one by 9% in supervised settings, this gap expands to nearly 40% in SSL.

The common assumption has been that the instance discrimination pretext task is too complex for low-capacity models, leading researchers to rely on computationally expensive Knowledge Distillation (KD) from pre-trained "teacher" models. This paper challenges that assumption, demonstrating that small models fail not because they cannot solve the task, but because they solve it "too well" in a way that hinders semantic generalization.

The "Easy Task" Paradox and Over-clustering

A key finding of this work is the "Easy Task" paradox: small models often achieve higher accuracy on the pretext task (instance discrimination) than their larger counterparts. Despite this, their linear probing performance on ImageNet is significantly worse.

Diagnostic metrics—including alignment, uniformity, and intra-class alignment—reveal a phenomenon termed over-clustering. Small models map augmented versions of the same image to nearly identical points in the feature space (perfect alignment) but fail to group semantically similar images together. This results in a representation space characterized by extremely tight, isolated points rather than meaningful semantic clusters.

t-SNE visualization comparing ResNet50 and MobileNetV3_large feature spaces, highlighting the over-clustering phenomenon in smaller models.

Proposed Recipe for Small SSL

To mitigate over-clustering and improve generalization without distillation signals, the authors propose a specific set of architectural and hyperparameter adjustments:

  1. Lower Temperature ($\tau$): Reducing the temperature (e.g., to 0.05) helps small models converge more effectively by sharpening the contrastive distribution.
  2. Wider MLP Projectors: Increasing the width of the projection head (e.g., doubling the hidden layer size) prevents an "Information Bottleneck" that might otherwise discard semantically useful features.
  3. Stronger Augmentation: Enhanced color jitter and blur prevent the model from relying on low-level "shortcuts" like color histograms to solve the instance discrimination task.
  4. Initialization Warm-up: A brief period of distillation (as few as 2 epochs) can help the model escape poor local minima, though the goal is to minimize reliance on teacher signals.

Experimental Results

The proposed "recipe" yields substantial improvements across a variety of small architectures, including ResNet18, MobileNetV3, EfficientNet-B0, and DeiT-Tiny. On ImageNet linear probing, these modifications boost performance by over 14% on average without any long-term distillation.

Comparison of Top-1 accuracy on ImageNet for various small models using baseline SSL versus the authors' improved recipe.
Model Baseline SSL Improved SSL Gain
MobileNetV3-Large 32.2% 48.7% +16.5%
MobileNetV3-Small 26.8% 41.3% +14.5%
DeiT-Tiny 23.4% 38.6% +15.2%
EfficientNet-B0 42.1% 55.9% +13.8%

Beyond ImageNet, these improvements generalize to transfer learning tasks on datasets such as CIFAR-10, CIFAR-100, and Caltech-101, significantly outperforming standard SSL baselines.

Conclusion

This research demonstrates that small models are capable of effective self-supervised learning without the need for heavy distillation signals. By identifying and addressing the issue of over-clustering through strategic hyperparameter and architectural choices, the performance gap between large and small SSL models can be significantly narrowed. Future work remains to bridge the remaining gap to distillation-based methods and to further explore the theoretical relationship between model capacity and contrastive temperature.

Summary updated

BibTeX
@inproceedings{shi2022efficacy,
  author = {Haizhou Shi and Youcai Zhang and Siliang Tang and Wenjie Zhu and Yaqian Li and Yandong Guo and Yueting Zhuang},
  title = {{On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i2.20120},
}
Figure