Abstract
Traditional supervised learning methods are hitting a bottleneck because of their dependency on expensive manually labeled data and their weaknesses such as limited generalization ability and vulnerability to adversarial attacks. A promising alternative, self-supervised learning, as a type of unsupervised learning, has gained popularity because of its potential to learn effective data representations without manual labeling. Among self-supervised learning algorithms, contrastive learning has achieved state-of-the-art performance in several fields of research. This literature review aims to provide an up-to-date analysis of the efforts of researchers to understand the key components and the limitations of self-supervised learning.
Introduction
The reliance on massive, human-annotated datasets like ImageNet has long been a double-edged sword for computer vision. While it fueled the deep learning revolution, it also created models that are often brittle and expensive to train. Self-supervised learning (SSL) seeks to overcome this by deriving supervisory signals directly from the data itself through "pretext tasks." Early pretext tasks, such as predicting image rotation or solving jigsaw puzzles, were "covariant"—the model's internal representation had to change in response to the transformation to solve the task. However, this often led the model to focus on low-level geometric features rather than the high-level semantic information needed for downstream tasks like object detection.
The Shift to Pretext-Invariance
A major breakthrough in SSL came with the shift toward Pretext-Invariant Representation Learning (PIRL). Instead of training a model to recognize a transformation, PIRL encourages the network to produce a representation that is invariant to it. In this framework, an image and its transformed version (e.g., a cropped or shuffled version) are treated as a "positive pair" that should have similar representations in the latent space.
As shown in Figure 1(b), this invariant approach significantly outperforms traditional covariant methods like Jigsaw puzzles. While both methods perform similarly in early layers (conv1, res2), PIRL maintains and even increases its accuracy in deeper layers (res4, res5), whereas the performance of covariant methods tends to drop off as the features become too specialized to the pretext task.
Landmark Contrastive Frameworks
Several key algorithms have defined the current state of contrastive learning:
- SimCLR: This Siamese-like framework uses the NT-Xent (Normalized Temperature-scaled Cross Entropy) loss. It takes an image, generates two augmented views, and optimizes the network to pull these views together while pushing all other images in the batch away. A critical discovery in SimCLR was the role of the "projector head"—a small MLP after the main encoder. Researchers found that the representation before this head is much better for downstream tasks, as the head tends to discard information that is not strictly necessary for the contrastive loss.
- MoCHi (Mixing of Contrastive Hard Negatives): Contrastive learning relies on "negative" samples to prevent the model from collapsing into a single point. MoCHi improves efficiency by synthesizing "hard negatives"—samples that are difficult for the model to distinguish—by mixing features in the latent space. This forces the model to learn finer semantic boundaries.
- LooC (Leave-one-out Contrastive Learning): To mitigate the risk of becoming too invariant (e.g., losing color information that might be useful later), LooC creates multiple embedding spaces. Each space is invariant to all but one specific augmentation, preserving a diverse set of features.
Theoretical Insights: Alignment and Uniformity
Beyond specific algorithms, researchers have identified two core properties that define a high-quality representation:
- Alignment: Similar samples should be mapped to nearby points on a hypersphere.
- Uniformity: The representations should be distributed as uniformly as possible on the hypersphere to preserve the maximal amount of information about the data distribution.
Limitations and Future Directions
Despite achieving performance that rivals supervised learning on benchmarks like ImageNet, contrastive learning still faces challenges. Most notably, it suffers from augmentation bias. Because humans must manually select which transformations (like cropping or color jittering) are "safe," the learning process is still implicitly supervised. If a downstream task requires information that was discarded during the "invariant" training phase (e.g., color for flower classification), the model will underperform.
Future research is moving toward Generalized Contrastive Learning, which aims to balance invariance with the preservation of useful covariant features, ensuring that self-supervised models can generalize across a wider variety of domains, from medical imaging to video analysis, without extensive manual tuning.
Summary updated