Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap

Weiyang Liu, Longhui Yu, Adrian Weller, Bernhard Schölkopf

Abstract

The neural collapse (NC) phenomenon describes an underlying geometric symmetry for deep neural networks, where both deeply learned features and classifiers converge to a simplex equiangular tight frame. It has been shown that both cross-entropy loss and mean square error can provably lead to NC. We remove NC's key assumption on the feature dimension and the number of classes, and then present a generalized neural collapse (GNC) hypothesis that effectively subsumes the original NC. Inspired by how NC characterizes the training target of neural networks, we decouple GNC into two objectives: minimal intra-class variability and maximal inter-class separability. We then use hyperspherical uniformity (which characterizes the degree of uniformity on the unit hypersphere) as a unified framework to quantify these two objectives. Finally, we propose a general objective -- hyperspherical uniformity gap (HUG), which is defined by the difference between inter-class and intra-class hyperspherical uniformity. HUG not only provably converges to GNC, but also decouples GNC into two separate objectives. Unlike cross-entropy loss that couples intra-class compactness and inter-class separability, HUG enjoys more flexibility and serves as a good alternative loss function. Empirical results show that HUG works well in terms of generalization and robustness.

Introduction

Neural Collapse (NC) is a phenomenon observed at the terminal phase of training where features of each class collapse into a single point, and these points arrange themselves into a perfectly symmetric Simplex Equiangular Tight Frame (ETF). However, the original NC theory requires the feature dimension ($d$) to be at least the number of classes minus one ($C-1$). This condition is often violated in large-scale real-world applications, such as face recognition. Additionally, standard loss functions like Cross-Entropy (CE) couple intra-class compactness and inter-class separability, limiting the ability to optimize these objectives independently.

Generalized Neural Collapse (GNC)

This work introduces GNC to address scenarios where $d < C-1$. In such cases, features and classifiers are hypothesized to converge toward a state of "hyperspherical uniformity," spreading as evenly as possible on the unit hypersphere. GNC acts as a universal law, reducing to the original NC when the dimensionality constraint is met.

2D feature visualization for 3 classes (left) and 10 classes (right), showing how features spread uniformly on the hypersphere.

Hyperspherical Uniformity Gap (HUG)

To leverage GNC, we propose the Hyperspherical Uniformity Gap (HUG) loss. HUG quantifies the degree of uniformity on the unit hypersphere and decouples the training target into two distinct goals: minimal intra-class variability and maximal inter-class separability. The HUG objective is defined as the difference between inter-class and intra-class hyperspherical uniformity, allowing for independent scaling of these components through hyperparameters $\alpha$ and $\beta$.

Evolution of inter-class separability metrics during training on MNIST for different class counts.

Key Results and Insights

Empirical evaluations across Computer Vision (CIFAR, ImageNet) and NLP (BERT fine-tuning) benchmarks highlight several advantages of the HUG framework:

  • Improved Generalization: On CIFAR-100, MHE-HUG achieved a 23.50% error rate, outperforming Cross-Entropy's 24.90%.
  • Enhanced Robustness: HUG significantly improved adversarial robustness, maintaining 15.24% accuracy under PGD attacks on CIFAR-10, compared to 7.94% for CE.
  • Long-Tailed Learning: HUG outperformed CE by nearly 2% in imbalanced settings, demonstrating the power of decoupled optimization.
  • Flatter Minima: Visualization of the loss landscape suggests that HUG leads to flatter local minima, which is strongly correlated with better generalization.

Conclusion

By generalizing Neural Collapse and introducing the HUG loss, this research provides a more flexible and robust framework for deep learning optimization. While HUG introduces new hyperparameters, its ability to decouple key geometric objectives offers a powerful alternative to traditional loss functions, especially in constrained or imbalanced environments.

Summary updated

Abstract

The neural collapse (NC) phenomenon describes an underlying geometric symmetry for deep neural networks, where both deeply learned features and classifiers converge to a simplex equiangular tight frame. It has been shown that both cross-entropy loss and mean square error can provably lead to NC. We remove NC's key assumption on the feature dimension and the number of classes, and then present a generalized neural collapse (GNC) hypothesis that effectively subsumes the original NC. Inspired by how NC characterizes the training target of neural networks, we decouple GNC into two objectives: minimal intra-class variability and maximal inter-class separability. We then use hyperspherical uniformity (which characterizes the degree of uniformity on the unit hypersphere) as a unified framework to quantify these two objectives. Finally, we propose a general objective -- hyperspherical uniformity gap (HUG), which is defined by the difference between inter-class and intra-class hyperspherical uniformity. HUG not only provably converges to GNC, but also decouples GNC into two separate objectives. Unlike cross-entropy loss that couples intra-class compactness and inter-class separability, HUG enjoys more flexibility and serves as a good alternative loss function. Empirical results show that HUG works well in terms of generalization and robustness.
BibTeX
@inproceedings{liu2023generalizing,
  author = {Weiyang Liu and Longhui Yu and Adrian Weller and Bernhard Schölkopf},
  title = {{Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=inU2quhGdNU},
}
Figure