In traditional supervised learning, a model learns to recognize a "dog" because a human has explicitly labeled thousands of images. In self-supervised learning, the goal is to learn these same rich representations without any human-provided labels. The core insight shared by recent breakthroughs is that we can use the data itself as its own supervision by enforcing view invariance.
If you take an image of a dog and crop it, flip it, or change its colors, the underlying semantic content—the "dog-ness"—remains the same. By forcing a neural network to produce similar mathematical representations for two different "views" of the same image, we can train it to ignore low-level pixel noise and focus on high-level concepts.
The Contrastive Approach: SimCLR
The SimCLR framework, introduced by Chen et al. (2020), formalizes this intuition using a contrastive learning objective. The mechanism is straightforward: for any given image in a batch, the system creates two different augmented versions (e.g., a random crop and a color distortion).
The model’s task is twofold:
- Maximize Agreement: Pull the representations of these two augmented views (the "positive pair") closer together in the latent space.
- Maximize Difference: Push the representation of the current image away from the representations of every other image in the batch (the "negative pairs").
Chen et al. demonstrated that two specific components are vital for this to work. First, a projection head—a small neural network placed after the main encoder—filters out information that is specific to the transformation (like the exact color palette) before the contrastive loss is applied. Second, the framework requires strong augmentations and large batch sizes. Without enough negative examples to "push" against, the model struggles to learn the boundaries between different objects.
Eliminating Negatives: BYOL
While SimCLR relies on comparing an image against many other images, Grill et al. (2020) proposed a different path with Bootstrap Your Own Latent (BYOL). They asked a provocative question: Can we learn these representations without using negative pairs at all?
The risk of removing negative pairs is "model collapse." If a model only tries to make two views of the same image similar, the easiest solution is to map every single image in the world to the exact same constant vector (e.g., a vector of all zeros). This satisfies the "similarity" requirement perfectly but conveys zero information.
BYOL avoids this collapse through an asymmetric architecture consisting of two networks: the online network and the target network.
- The online network learns to predict the representation of the target network.
- The target network is not trained via gradient descent. Instead, its weights are an exponential moving average (EMA) of the online network's weights.
By adding a "predictor" MLP to the online network and using the slowly-evolving EMA target, Grill et al. created a system where the target is always a slightly different, more stable version of the online network. This prevents the two networks from collapsing into a trivial solution, allowing the model to learn high-quality features using only positive pairs.
Why View Invariance Works
The success of both SimCLR and BYOL suggests that the "pretext task" of matching augmented views is a powerful proxy for understanding the visual world.
When we force a model to see a grayscale crop of a tail and a color-distorted crop of a snout as the "same" object, we are implicitly teaching it about object parts, textures, and shapes. The model learns that while pixels change under different lighting or framing, the identity of the object is invariant.
By the time these models are finished with this self-supervised "pre-training," they have developed a sophisticated visual vocabulary. When a small amount of labeled data is finally introduced, the models don't need to learn what a dog looks like from scratch; they simply need to learn that the cluster of features they already recognize should be mapped to the word "dog."