Dissecting Supervised Constrastive Learning

Florian Graf, Christoph D. Hofer, Marc Niethammer, Roland Kwitt

Abstract

Recent advances in deep learning have seen Supervised Contrastive (SC) learning emerge as a powerful alternative to standard Cross-Entropy (CE) loss. This study investigates the theoretical underpinnings of SC, proving that both CE and SC share a common global minimum: the collapse of class representations into the vertices of a regular simplex. Despite this shared destination, SC exhibits unique training dynamics, specifically a superlinear difficulty in fitting noisy labels, suggesting a robust implicit regularization mechanism.

Introduction

In supervised deep learning, minimizing Cross-Entropy (CE) loss over softmax scores has long been the standard. However, Supervised Contrastive (SC) learning has recently demonstrated superior robustness and representation quality. This paper addresses whether these two loss functions drive the encoder toward different geometric configurations in the representation space and why SC appears more resilient to label noise.

Methods

The authors employed a multi-faceted approach:

  1. Theoretical Analysis: Deriving lower bounds for CE and SC losses under an "ideal encoder" assumption.
  2. Geometric Characterization: Defining the $\rho$-sphere-inscribed regular simplex as the optimal state for class separation.
  3. Numerical Simulation: Using ResNet-18 on CIFAR-10/100 to observe real-world convergence.
  4. Label Corruption Experiments: Measuring the time required to fit datasets with varying levels of randomly flipped labels.

Key Insights: The Simplex Collapse

A central finding is that both CE and SC reach their global minimum when "Neural Collapse" occurs. In this state, all samples of a given class collapse to a single point, and these points form the vertices of a regular simplex on a hypersphere.

Figure 2: Visualization of the convergence process for Supervised Contrastive (SC) and Cross-Entropy (CE) losses toward a regular simplex configuration.

While CE treats samples independently, SC relies on batch-wise interactions, creating a balance of attraction and repulsion. The authors demonstrate that the average over all batches eventually leads to this simplex state. However, architectural constraints like ReLU activations in standard ResNets can prevent models from reaching the true mathematical optimum by forcing non-negative coordinates.

Key Results: Hardness of Memorization

A critical distinction between the two methods lies in their response to label noise. While CE can eventually "memorize" corrupted labels in a linear timeframe, SC exhibits a "lazy" behavior. The time required for SC to fit noisy data scales superlinearly, making it structurally difficult for the model to overfit to random labels.

Figure 4: Comparison of time to fit versus label corruption. SC shows a superlinear increase in training time as noise increases, unlike the relatively flat response of CE.

Empirical results on CIFAR-10 show that SC fails to reach zero training error when corruption exceeds 80%, whereas CE continues to memorize the noise. Furthermore, reaching a state closer to the theoretical simplex minimum correlates with improved generalization on test data.

Limitations and Future Work

The study assumes an infinitely flexible "ideal encoder," which ignores architectural constraints that restrict geometric configurations. Future research could explore the role of the "projection head" in SC and the specific stochastic gradient descent (SGD) dynamics that make SC harder to fit to noise.

Summary updated

Abstract

Minimizing cross-entropy over the softmax scores of a linear map composed with a high-capacity encoder is arguably the most popular choice for training neural networks on supervised learning tasks....
BibTeX
@inproceedings{graf2021dissecting,
  author = {Florian Graf and Christoph D. Hofer and Marc Niethammer and Roland Kwitt},
  title = {{Dissecting Supervised Constrastive Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=rg-gnmNC1gq},
}
Figure