Abstract
Supervised-contrastive loss (SCL) is an alternative to cross-entropy (CE) for classification tasks that makes use of similarities in the embedding space to allow for richer representations. In this work, we propose methods to engineer the geometry of these learnt feature embeddings by modifying the contrastive loss. In pursuit of adjusting the geometry we explore the impact of prototypes, fixed embeddings included during training to alter the final feature geometry. Specifically, through empirical findings, we demonstrate that the inclusion of prototypes in every batch induces the geometry of the learnt embeddings to align with that of the prototypes. We gain further insights by considering a limiting scenario where the number of prototypes far outnumber the original batch size. Through this, we establish a connection to cross-entropy (CE) loss with a fixed classifier and normalized embeddings. We validate our findings by conducting a series of experiments with deep neural networks on benchmark vision datasets.
Introduction
In deep learning, the final stages of training often lead to a phenomenon known as Neural Collapse (NC). In this state, the feature embeddings of different classes converge into a highly symmetric structure called a Simplex Equiangular Tight Frame (ETF). While this symmetry is desirable for generalization, it typically only emerges under ideal conditions: balanced datasets and standard Cross-Entropy (CE) loss.
When datasets are imbalanced—a common occurrence in real-world applications—this geometric symmetry breaks down, often degrading performance. While Supervised-Contrastive Loss (SCL) offers a powerful alternative to CE, its geometric behavior under class imbalance has remained difficult to control. This research addresses a fundamental question: Can we intentionally engineer the internal geometry of neural networks to maintain or even improve performance under challenging data distributions?
Methodology: Engineering Geometry with Prototypes
The researchers propose a novel modification to the SCL training process by introducing fixed, non-trainable "prototypes" for each class. By injecting $n_w$ copies of these fixed vectors into every training batch, the network is forced to align its feature embeddings with the pre-defined geometry of the prototypes.
This approach allows for "Geometry Engineering," where the final shape of the embedding space is dictated by design. Three specific geometries were explored:
- Simplex ETF: Enforcing perfect symmetry across all classes.
- Improved Minority Angles: Intentionally providing minority classes with larger angular separation to improve their distinguishability.
- Majority Collapse: Forcing majority classes toward a single point to focus the model's capacity on differentiating minority classes.
Mathematically, the authors demonstrate that as the number of prototypes ($n_w$) becomes sufficiently large, the SCL function converges to a variant of Cross-Entropy loss with a fixed classifier and normalized embeddings, providing a theoretical bridge between these two paradigms.
Experimental Results
The proposed method was validated using ResNet-18 on CIFAR-10 and its imbalanced variants (STEP-imbalanced CIFAR-10 with imbalance ratios R=10 and R=100).
Geometric Convergence
A key finding is the effectiveness of fixed anchors. Unlike vanilla SCL, which fails to achieve a symmetric ETF geometry under imbalance, the addition of prototypes (e.g., $n_w = 100$) induces nearly perfect convergence to the target geometry. This is visualized through Gram matrices, which represent the angles between class centers.
As shown in the Gram matrices, the proposed method produces significantly cleaner and more distinct class separations compared to both standard SCL and Balanced Contrastive Learning (BCL), even at high imbalance ratios (R=100).
Computational Efficiency
Importantly, the researchers found that increasing the number of prototypes does not significantly impact computational overhead. This makes the "large $n_w$" limit a practical and viable strategy for training robust models.
Discussion and Future Work
This work demonstrates that fixed prototypes are a sufficient and efficient mechanism for guiding neural network geometry. The ability to design asymmetric geometries (like "Improved Minority Angles") opens new avenues for handling long-tailed data distributions.
However, several questions remain. While the method successfully controls the geometry of embeddings, the direct translation of specific geometric configurations to top-1 test accuracy across diverse datasets requires further benchmarking. Future research will focus on identifying optimal non-symmetric geometries for specific real-world imbalance scenarios to maximize generalization performance.
Summary updated