Understanding Contrastive Learning Requires Incorporating Inductive Biases

Nikunj Saunshi, Jordan T. Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham M. Kakade, Akshay Krishnamurthy

Introduction

Contrastive learning (CL) has become a cornerstone of modern self-supervised AI, allowing models to learn powerful representations from unlabeled data by ensuring that different "views" (augmentations) of the same input stay close together in feature space, while views of different inputs are pushed apart.

Despite its empirical success, our theoretical understanding of CL is lagging. Existing theories often assume that for CL to work, augmentations of the same object must have significant overlap. However, this paper argues that these "function-class-agnostic" theories fail to explain reality. They cannot account for why a ResNet might succeed where a Vision Transformer (ViT) fails on the exact same data, or why CL works even when augmentations are "disjoint" (having zero overlap)—a scenario where previous theories offer only meaningless guarantees. The authors reveal that current theoretical explanations are incomplete because they ignore the "inductive biases"—the inherent preferences—of model architectures and training algorithms.

Methods and Datasets

The authors combine rigorous mathematical proofs with targeted experiments to expose the gaps in current CL theory:

  • Lower Bound Proofs: They mathematically prove that if you ignore the specific architecture (the function class), you can always find "spurious" solutions that perfectly minimize contrastive loss but fail miserably at downstream tasks.
  • New Transfer Bounds: They introduce new theoretical bounds specifically for linear representations that incorporate the properties of the function class, showing how the right inductive bias can overcome a lack of augmentation overlap.
  • Probing Disjointness: By training a model to identify which exact image an augmentation came from, they show that standard augmentations are much closer to being disjoint than previously assumed.
  • Adversarial "Label-Orthogonal" Training: They created pathological training setups to demonstrate the "brittleness of transfer," where low contrastive loss does not always translate to high downstream accuracy.

The study uses a mix of synthetic and real-world benchmarks, including a custom Boolean Hypercube dataset to illustrate theoretical points, CIFAR-10 for testing image architectures (ResNet-18, ViT, MLP-Mixer), and AG News for text classification (Bag-of-Words, GRU, Transformer).

Key Insights

  • Architecture is Destiny: The success of contrastive learning isn't just about the loss function; it is heavily dependent on the architecture. For example, ResNets naturally ignore high-frequency noise that might lead a more flexible model like a ViT to find a "shortcut" solution.
  • The Overlap Myth: Contrary to popular belief, significant overlap between augmentations isn't strictly necessary. CL can work in the "disjoint regime" if the model's inductive bias prevents it from learning trivial identities.
  • More Loss Isn't Always Better: In some cases, a model with higher contrastive loss can actually have better downstream accuracy because its architecture prevents it from over-optimizing on useless features.

Key Results

The paper's findings are best summarized by the stark differences in how architectures handle the same task:

1. The Hypercube Example Using a synthetic high-dimensional dataset, the authors demonstrate that a simple linear model achieves 99.5% accuracy despite having a higher contrastive loss than a Multi-Layer Perceptron (MLP), which only reaches ~74%. This proves that the "simpler" function class has a better inductive bias for this specific task, preventing it from learning spurious representations.

Table 1: Results on the hypercube example showing that lower loss doesn't always mean better accuracy.

2. The "Hash Pixel" Test To test how models handle disjoint augmentations, the authors added unique "hash pixels" to images, making the augmentations perfectly disjoint. As shown in the bottom right of the figure below, the accuracy of ViT and MLP-Mixer models collapsed to near-random levels, while the ResNet-18 remained largely unaffected. The ResNet's convolutional structure provided an inductive bias that ignored the hash pixels, whereas the other architectures memorized them to minimize contrastive loss.

Figure 3: Transfer plots for CIFAR-10 showing how different architectures respond to various augmentations, including the "hash pixel" test.

3. Text Domain Surprises In text tasks like AG News, a simple Bag-of-Words (BoW) model often outperformed complex Transformers on downstream tasks despite having worse contrastive loss. The BoW's order-invariance acted as a beneficial inductive bias, further proving that architecture plays a pivotal role across different modalities.

Limitations and Future Work

The primary limitation of this work is that the new, tighter theoretical bounds are currently limited to linear representations. While the empirical experiments strongly suggest these principles apply to deep neural networks, proving it mathematically for complex architectures like ResNets or Transformers remains an open challenge.

The authors suggest that the next frontier for CL theory is to precisely characterize the inductive biases of deep architectures and training procedures (like SGD vs. Adam) to provide a truly predictive framework for self-supervised learning.

Summary updated

Abstract

Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of differen...
BibTeX
@inproceedings{saunshi2022understanding,
  author = {Nikunj Saunshi and Jordan T. Ash and Surbhi Goel and Dipendra Misra and Cyril Zhang and Sanjeev Arora and Sham M. Kakade and Akshay Krishnamurthy},
  title = {{Understanding Contrastive Learning Requires Incorporating Inductive Biases}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=kSin1WyiDM},
}
Figure