Understanding Contrastive Learning Requires Incorporating Inductive Biases

Nikunj Saunshi, Jordan T. Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham M. Kakade, Akshay Krishnamurthy

TL;DR

Current theoretical explanations for contrastive learning's success often overlook the crucial role of model architecture (inductive biases) and training algorithms. This paper demonstrates that these factors are essential, showing that ignoring them can lead to vacuous theoretical guarantees and brittle real-world performance. The authors provide theoretical and experimental evidence that incorporating inductive biases is key to understanding and improving contrastive learning.

What Problem Does This Paper Address?

Contrastive learning has achieved remarkable success in self-supervised representation learning, enabling models to learn powerful features from unlabeled data. However, a solid theoretical understanding of why it works so well, especially across diverse tasks and data distributions, has lagged behind its empirical achievements. Existing theoretical frameworks often focus on the properties of data augmentations and the resulting contrastive loss value, but largely ignore the "inductive biases" of the function class (i.e., the model architecture) and the training algorithm. This oversight can lead to incomplete explanations and potentially misleading guarantees, as demonstrated by the paper's findings that these factors significantly impact downstream performance.

Methods

The paper combines theoretical analysis with extensive empirical experiments.

Theoretical Contributions: * Formalizing the problem: The authors define contrastive loss and downstream classification error, and introduce a framework for analyzing "transfer bounds" that relate performance on the contrastive task to downstream performance. * Identifying limitations of existing theories: They show that function-class-agnostic analyses (which ignore inductive biases) can lead to vacuous guarantees, especially in "disjoint augmentation" regimes where augmentations from different classes have no overlap. * Developing function-class-dependent analysis: The core theoretical contribution is a new analysis for linear representation classes. This analysis incorporates the function class and demonstrates how it allows for more meaningful guarantees under weaker assumptions on augmentations. * Lower bounds: The paper establishes lower bounds for function-class-agnostic analyses, highlighting their limitations.

Empirical Contributions: * Synthetic experiments: A simplified "Boolean hypercube" example is used to illustrate the phenomena of function class sensitivity and brittle transfer. * Image domain experiments: CIFAR-10 is used with SimCLR augmentations to show how different architectures (ResNet, ViT, MLP-Mixer) and training configurations (SGD vs. Adam) exhibit varying transfer performance. * Text domain experiments: The AG News dataset is used to evaluate Bag-of-Words, GRU, and Transformer models, demonstrating similar phenomena in a different modality. * Visualizations: 2D representations are visualized to provide intuition about why certain models succeed or fail on downstream tasks.

Datasets

  • CIFAR-10: Used for image classification experiments with SimCLR augmentations.
  • AG News: Used for text classification experiments.

Key Insights and Ideas

  • Inductive biases are crucial: The success of contrastive learning is not solely determined by the contrastive loss value or augmentation properties. The choice of model architecture (function class) and training algorithm plays a significant role in how well representations transfer to downstream tasks.
  • Disjoint augmentations are problematic for agnostic theories: When augmentations from different classes have no overlap, existing theoretical analyses that ignore function class biases can become vacuous, failing to explain why contrastive learning might still succeed with appropriate models.
  • Function class sensitivity: Different architectures can exhibit drastically different downstream performance even when trained with the same augmentations and achieving similar contrastive losses. This highlights the importance of inductive biases in guiding representations towards useful features.
  • Brittleness of transfer: Minimizing contrastive loss can sometimes lead to poor downstream performance, especially for more expressive models or under specific augmentation regimes. This "brittleness" is a key phenomenon the paper aims to explain.
  • Linear representations offer a tractable theoretical ground: The paper shows that by focusing on linear representation classes, it's possible to develop theoretical guarantees that incorporate inductive biases and provide more meaningful insights.

Key Results

  • Synthetic Hypercube Example:
    • Demonstrated the existence of "perfect" and "spurious" representations that both minimize contrastive loss but have vastly different downstream accuracy.
  • Showed that linear models achieved near-perfect downstream accuracy, while more complex MLPs were sensitive to training configurations.
  • CIFAR-10 Experiments:
    • Removal of certain augmentations (e.g., random cropping) accentuated performance differences between ResNet, ViT, and MLP-Mixer.
  • In a "hash pixels" setting designed to create disjoint augmentations, ResNet performed well despite a higher contrastive loss, while ViT and MLP-Mixer showed transfer collapse.
  • Experiments confirmed that augmentations in standard settings (like SimCLR) are close to the disjoint regime, yet contrastive learning still succeeds, underscoring the role of inductive biases.
  • AG News Experiments:
    • BoW models consistently performed well downstream, while GRU and Transformer models showed brittle transfer across different augmentations.
  • Visualizations of 2D representations revealed that Transformer models learned representations that were augmentation invariant but not linearly separable, explaining their poor downstream performance.

Limitations and Future Work

  • Theoretical focus on linear representations: While the paper provides theoretical guarantees for linear representations, extending these to more complex, deep neural network architectures remains an open challenge.
  • Incorporating training algorithms: The current theoretical analysis primarily focuses on function class bias. Incorporating the influence of training algorithms (like SGD, Adam) into theoretical bounds is a promising direction.
  • Algorithmic implications: The paper's diagnostic nature identifies gaps in understanding. Converting these insights into algorithmic approaches for more robust contrastive learning is a key avenue for future work.
  • Visualizations for augmentation selection: The authors hope that visualizations of contrastive loss vs. downstream performance can aid in selecting more robust augmentations.

Summary updated

Abstract

Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs. Recent attempts to theoretically explain the success of contrastive learning on downstream classification tasks prove guarantees depending on properties of {\em augmentations} and the value of {\em contrastive loss} of representations. We demonstrate that such analyses, that ignore {\em inductive biases} of the function class and training algorithm, cannot adequately explain the success of contrastive learning, even {\em provably} leading to vacuous guarantees in some settings. Extensive experiments on image and text domains highlight the ubiquity of this problem -- different function classes and algorithms behave very differently on downstream tasks, despite having the same augmentations and contrastive losses. Theoretical analysis is presented for the class of linear representations, where incorporating inductive biases of the function class allows contrastive learning to work with less stringent conditions compared to prior analyses.
BibTeX
@inproceedings{saunshi2022understanding,
  author = {Nikunj Saunshi and Jordan T. Ash and Surbhi Goel and Dipendra Misra and Cyril Zhang and Sanjeev Arora and Sham M. Kakade and Akshay Krishnamurthy},
  title = {{Understanding Contrastive Learning Requires Incorporating Inductive Biases}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=kSin1WyiDM},
}
Figure