Understanding the Benefits of SimCLR Pre-Training in Two-Layer Convolutional Neural Networks

Han Zhang, Yuan CaoSubmission to ICLR 2025 · Rejected

Abstract

SimCLR is one of the most popular contrastive learning methods for vision tasks. It pre-trains deep neural networks based on a large amount of unlabeled data by teaching the model to distinguish between positive and negative pairs of augmented images. It is believed that SimCLR can pre-train a deep neural network to learn efficient representations that can lead to a better performance of future supervised fine-tuning. Despite its effectiveness, our theoretical understanding of the underlying mechanisms of SimCLR is still limited. In this paper, we theoretically introduce a case study of the SimCLR method. Specifically, we consider training a two-layer convolutional neural network (CNN) to learn a toy image data model. We show that, under certain conditions on the number of labeled data, SimCLR pre-training combined with supervised fine-tuning achieves almost optimal test loss. Notably, the label complexity for SimCLR pre-training is far less demanding compared to direct training on supervised data. Our analysis sheds light on the benefits of SimCLR in learning with fewer labels.
BibTeX
@inproceedings{zhang2026understanding,
  author = {Han Zhang and Yuan Cao},
  title = {{Understanding the Benefits of SimCLR Pre-Training in Two-Layer Convolutional Neural Networks}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2026},
  url = {https://openreview.net/forum?id=HN1ItuXq5M},
}
Figure