Abstract
We propose SigCLR: Sigmoid Contrastive Learning of Visual Representations. SigCLR utilizes the logistic loss that only operates on pairs and does not require a global view as in the cross-entropy loss used in SimCLR. We show that logistic loss shows competitive performance on CIFAR-10, CIFAR-100, and Tiny-IN compared to other established SSL objectives. Our findings verify the importance of learnable bias as in the case of SigLUP, however, it requires a fixed temperature as in the SimCLR to excel. Overall, SigCLR is a promising replacement for the SimCLR which is ubiquitous and has shown tremendous success in various domains.
Introduction
Self-supervised learning (SSL) has become a cornerstone of modern computer vision, with contrastive methods like SimCLR setting the standard for representation learning. These methods typically rely on the InfoNCE loss, a softmax-based objective that requires a global normalization factor calculated across the entire batch. This global dependency introduces computational bottlenecks in distributed training environments and often necessitates large batch sizes to ensure stable gradients.
SigCLR addresses these limitations by adapting the sigmoid-based loss—recently popularized in multi-modal models like SigLIP—for vision-only SSL. By reframing contrastive learning as a series of independent pairwise binary classifications, SigCLR eliminates the need for global normalization, offering a more scalable and efficient alternative for visual representation learning.
Methodology
SigCLR follows the general architecture of SimCLR, utilizing an encoder followed by a projection head. The primary innovation lies in the replacement of the InfoNCE loss with a pairwise sigmoid loss formulation.
The key technical components of SigCLR include:
- Pairwise Sigmoid Loss: Instead of normalizing across the whole batch, SigCLR treats every pair of images independently, simplifying the optimization objective.
- Learnable Bias: A critical addition is a learnable bias term ($b$). This term is essential for managing the extreme imbalance between positive and negative pairs during the initial stages of training, allowing the model to navigate the sparse signal effectively.
- Fixed Temperature: Interestingly, while multi-modal models like SigLIP benefit from a learnable temperature, the authors found that a fixed temperature (consistent with SimCLR) yields superior results for vision-only pretraining.
- Distributed Efficiency: The pairwise nature of the loss enables a "chunked" implementation that avoids memory-intensive "all-gather" operations, making it highly suitable for large-scale distributed training.
Experimental Results
The authors evaluated SigCLR across several standard benchmarks, including CIFAR-10, CIFAR-100, and Tiny-ImageNet (Tiny-IN). Performance was measured using the linear evaluation protocol, where a linear classifier is trained on top of the frozen backbone.
As shown in the results above, SigCLR consistently matches or exceeds the performance of a reproduced SimCLR baseline (SimCLR*) and remains competitive with other established SSL methods such as BYOL and MoCo-V2. Notably, SigCLR demonstrates significant gains on CIFAR-100 and Tiny-IN.
Furthermore, SigCLR exhibits robustness to batch size variations. In experiments ranging from batch sizes of 64 to 1024, SigCLR maintained high performance, often outperforming SimCLR* at smaller batch sizes. This suggests that the sigmoid loss is less dependent on the batch providing a perfectly representative distribution of negatives compared to softmax-based alternatives.
Discussion and Insights
A central insight of this work is the "secret sauce" of the learnable bias. Without this bias, the sigmoid loss performs poorly, particularly as the temperature increases. The bias allows the model to effectively "ignore" the overwhelming number of negative samples early in training, focusing the optimization on the relevant contrastive signals.
Additionally, the study highlights a divergence between vision-only and multi-modal SSL dynamics. While learnable temperature is a standard feature in models like SigLIP, it was found to be detrimental in the context of SigCLR. This suggests that the alignment of two distinct modalities (image and text) involves different optimization challenges than the self-alignment of augmented visual views.
Conclusion
SigCLR presents a compelling alternative to the ubiquitous SimCLR framework. By leveraging a pairwise sigmoid loss with a learnable bias, it achieves state-of-the-art performance while improving scalability and reducing computational overhead. Future research will explore the application of SigCLR to larger datasets like ImageNet-1k and its performance with more diverse architectures such as Vision Transformers.
Summary updated