Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss

Jaidev Gill, Vala Vakilian, Christos Thrampoulidis

Abstract

Supervised-contrastive loss (SCL) is an alternative to cross-entropy (CE) for classification tasks that makes use of similarities in the embedding space to allow for richer representations. In this work, we propose methods to engineer the geometry of these learnt feature embeddings by modifying the contrastive loss. In pursuit of adjusting the geometry we explore the impact of prototypes, fixed embeddings included during training to alter the final feature geometry. Specifically, through empirical findings, we demonstrate that the inclusion of prototypes in every batch induces the geometry of the learnt embeddings to align with that of the prototypes. We gain further insights by considering a limiting scenario where the number of prototypes far outnumber the original batch size. Through this, we establish a connection to cross-entropy (CE) loss with a fixed classifier and normalized embeddings. We validate our findings by conducting a series of experiments with deep neural networks on benchmark vision datasets.

Introduction

In deep learning, the final stages of training often lead to a phenomenon known as Neural Collapse (NC). In this state, the feature embeddings of different classes converge into a highly symmetric structure called a Simplex Equiangular Tight Frame (ETF). While this symmetry is desirable for generalization, it typically only emerges under ideal conditions: balanced datasets and standard Cross-Entropy (CE) loss.

When datasets are imbalanced—a common occurrence in real-world applications—this geometric symmetry breaks down, often degrading performance. While Supervised-Contrastive Loss (SCL) offers a powerful alternative to CE, its geometric behavior under class imbalance has remained difficult to control. This research addresses a fundamental question: Can we intentionally engineer the internal geometry of neural networks to maintain or even improve performance under challenging data distributions?

Methodology: Engineering Geometry with Prototypes

The researchers propose a novel modification to the SCL training process by introducing fixed, non-trainable "prototypes" for each class. By injecting $n_w$ copies of these fixed vectors into every training batch, the network is forced to align its feature embeddings with the pre-defined geometry of the prototypes.

This approach allows for "Geometry Engineering," where the final shape of the embedding space is dictated by design. Three specific geometries were explored:

  1. Simplex ETF: Enforcing perfect symmetry across all classes.
  2. Improved Minority Angles: Intentionally providing minority classes with larger angular separation to improve their distinguishability.
  3. Majority Collapse: Forcing majority classes toward a single point to focus the model's capacity on differentiating minority classes.
Visualization of target geometries: Simplex ETF, Improved Minority Angles, and Majority Collapse across different imbalance ratios (R).

Mathematically, the authors demonstrate that as the number of prototypes ($n_w$) becomes sufficiently large, the SCL function converges to a variant of Cross-Entropy loss with a fixed classifier and normalized embeddings, providing a theoretical bridge between these two paradigms.

Experimental Results

The proposed method was validated using ResNet-18 on CIFAR-10 and its imbalanced variants (STEP-imbalanced CIFAR-10 with imbalance ratios R=10 and R=100).

Geometric Convergence

A key finding is the effectiveness of fixed anchors. Unlike vanilla SCL, which fails to achieve a symmetric ETF geometry under imbalance, the addition of prototypes (e.g., $n_w = 100$) induces nearly perfect convergence to the target geometry. This is visualized through Gram matrices, which represent the angles between class centers.

Gram matrix comparison between vanilla SCL, Balanced Contrastive Learning (BCL), and the proposed SCL + 100 Prototypes method.

As shown in the Gram matrices, the proposed method produces significantly cleaner and more distinct class separations compared to both standard SCL and Balanced Contrastive Learning (BCL), even at high imbalance ratios (R=100).

Computational Efficiency

Importantly, the researchers found that increasing the number of prototypes does not significantly impact computational overhead. This makes the "large $n_w$" limit a practical and viable strategy for training robust models.

Discussion and Future Work

This work demonstrates that fixed prototypes are a sufficient and efficient mechanism for guiding neural network geometry. The ability to design asymmetric geometries (like "Improved Minority Angles") opens new avenues for handling long-tailed data distributions.

However, several questions remain. While the method successfully controls the geometry of embeddings, the direct translation of specific geometric configurations to top-1 test accuracy across diverse datasets requires further benchmarking. Future research will focus on identifying optimal non-symmetric geometries for specific real-world imbalance scenarios to maximize generalization performance.

Summary updated

Abstract

Supervised-contrastive loss (SCL) is an alternative to cross-entropy (CE) for classification tasks that makes use of similarities in the embedding space to allow for richer representations. In this work, we propose methods to engineer the geometry of these learnt feature embeddings by modifying the contrastive loss. In pursuit of adjusting the geometry we explore the impact of prototypes, fixed embeddings included during training to alter the final feature geometry. Specifically, through empirical findings, we demonstrate that the inclusion of prototypes in every batch induces the geometry of the learnt embeddings to align with that of the prototypes. We gain further insights by considering a limiting scenario where the number of prototypes far outnumber the original batch size. Through this, we establish a connection to cross-entropy (CE) loss with a fixed classifier and normalized embeddings. We validate our findings by conducting a series of experiments with deep neural networks on benchmark vision datasets.
BibTeX
@inproceedings{gill2024engineering,
  author = {Jaidev Gill and Vala Vakilian and Christos Thrampoulidis},
  title = {{Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447379},
}
Figure