Learning Visual Representations through Image Augmentations

This direction is merged into another direction: dimensionality limitation (2D to 3D extension). Its last version stays readable and citable here.

Version 1 of 12 papers17 conceptsfrozen 2026-10-04

In traditional supervised learning, a model learns to recognize a "dog" because a human has explicitly labeled thousands of images. In self-supervised learning, the goal is to learn these same rich representations without any human-provided labels. The core insight shared by recent breakthroughs is that we can use the data itself as its own supervision by enforcing view invariance.

If you take an image of a dog and crop it, flip it, or change its colors, the underlying semantic content—the "dog-ness"—remains the same. By forcing a neural network to produce similar mathematical representations for two different "views" of the same image, we can train it to ignore low-level pixel noise and focus on high-level concepts.

The Contrastive Approach: SimCLR

The SimCLR framework, introduced by Chen et al. (2020), formalizes this intuition using a contrastive learning objective. The mechanism is straightforward: for any given image in a batch, the system creates two different augmented versions (e.g., a random crop and a color distortion).

The model’s task is twofold:

  1. Maximize Agreement: Pull the representations of these two augmented views (the "positive pair") closer together in the latent space.
  2. Maximize Difference: Push the representation of the current image away from the representations of every other image in the batch (the "negative pairs").

Chen et al. demonstrated that two specific components are vital for this to work. First, a projection head—a small neural network placed after the main encoder—filters out information that is specific to the transformation (like the exact color palette) before the contrastive loss is applied. Second, the framework requires strong augmentations and large batch sizes. Without enough negative examples to "push" against, the model struggles to learn the boundaries between different objects.

Eliminating Negatives: BYOL

While SimCLR relies on comparing an image against many other images, Grill et al. (2020) proposed a different path with Bootstrap Your Own Latent (BYOL). They asked a provocative question: Can we learn these representations without using negative pairs at all?

The risk of removing negative pairs is "model collapse." If a model only tries to make two views of the same image similar, the easiest solution is to map every single image in the world to the exact same constant vector (e.g., a vector of all zeros). This satisfies the "similarity" requirement perfectly but conveys zero information.

BYOL avoids this collapse through an asymmetric architecture consisting of two networks: the online network and the target network.

  • The online network learns to predict the representation of the target network.
  • The target network is not trained via gradient descent. Instead, its weights are an exponential moving average (EMA) of the online network's weights.

By adding a "predictor" MLP to the online network and using the slowly-evolving EMA target, Grill et al. created a system where the target is always a slightly different, more stable version of the online network. This prevents the two networks from collapsing into a trivial solution, allowing the model to learn high-quality features using only positive pairs.

Why View Invariance Works

The success of both SimCLR and BYOL suggests that the "pretext task" of matching augmented views is a powerful proxy for understanding the visual world.

When we force a model to see a grayscale crop of a tail and a color-distorted crop of a snout as the "same" object, we are implicitly teaching it about object parts, textures, and shapes. The model learns that while pixels change under different lighting or framing, the identity of the object is invariant.

By the time these models are finished with this self-supervised "pre-training," they have developed a sophisticated visual vocabulary. When a small amount of labeled data is finally introduced, the models don't need to learn what a dog looks like from scratch; they simply need to learn that the cluster of features they already recognize should be mapped to the word "dog."

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_learning_visual_representations_through__3c8ca1e5,
  title = {Learning Visual Representations through Image Augmentations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-visual-representations-through-image-augmentations/rdr_3c8ca1e5c7de46639b12ac8612ffe747},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_visual_representations_through__3c8ca1e5_v1,
  title = {Learning Visual Representations through Image Augmentations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-visual-representations-through-image-augmentations/rdr_3c8ca1e5c7de46639b12ac8612ffe747/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}