Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap

Weiyang Liu, Longhui Yu, Adrian Weller, Bernhard Schölkopf

TL;DR

This paper introduces Generalized Neural Collapse (GNC), extending the "Neural Collapse" phenomenon to scenarios where the feature dimension is smaller than the number of classes. To achieve this state, the authors propose the Hyperspherical Uniformity Gap (HUG), a novel loss function that decouples intra-class compactness from inter-class separability. This decoupling leads to better generalization, flatter loss landscapes, and significantly improved adversarial robustness compared to standard cross-entropy.

What Problem Does This Paper Address?

In deep learning, "Neural Collapse" (NC) describes a fascinating geometric symmetry that emerges at the end of training: class features collapse to their means, and those means arrange themselves into a perfectly symmetric structure called a Simplex Equiangular Tight Frame (ETF).

Geometric structures of Neural Collapse

However, NC has a major catch: it mathematically requires the feature dimension ($d$) to be at least as large as the number of classes minus one ($C-1$). In many real-world applications, like face recognition with 100k+ identities, we often use much smaller feature dimensions (e.g., $d=512$). Standard NC theory doesn't tell us what happens here. Furthermore, common loss functions like Cross-Entropy (CE) "couple" the goals of pushing classes apart and pulling same-class features together, making it hard to optimize them independently.

Methods

The authors tackle these issues through two main contributions:

1. Generalized Neural Collapse (GNC)

They hypothesize that even when $d < C-1$, features and classifiers still strive for a symmetric state characterized by hyperspherical uniformity—essentially spreading out as evenly as possible on a unit sphere, even if they can't form a perfect simplex.

2D feature visualization showing neural collapse

2. Hyperspherical Uniformity Gap (HUG)

They propose a new objective function defined as: $$\text{HUG} = \alpha \cdot (\text{Inter-class Uniformity}) - \beta \cdot (\text{Intra-class Uniformity})$$ * Inter-class term: Encourages class centers (proxies) to be maximally distant from each other. * Intra-class term: Encourages features within a class to collapse toward their respective class proxy.

The authors explore several ways to measure "uniformity," including Minimum Hyperspherical Energy (MHE) (inspired by electrons repelling each other on a sphere) and Maximum Gram Determinant (MGD) (maximizing the volume spanned by vectors).

Key Insights and Ideas

  • Decoupling is Powerful: By separating the "push" (inter-class) and "pull" (intra-class) forces, HUG allows for more flexible training. You can prioritize class separation without over-constraining feature compactness, or vice versa.
  • Static Proxies Work: Surprisingly, you don't necessarily need to "learn" class weights. If you initialize class proxies as points spread evenly on a sphere and freeze them, the network can still learn highly effective features. This could significantly reduce GPU memory for tasks with massive numbers of classes.
  • Flatter Minima: Visualization of the loss landscape shows that HUG produces much flatter local minima than Cross-Entropy. In deep learning theory, flatter minima are strongly associated with better generalization to unseen data.
  • CE is an Implicit HUG: The authors prove mathematically that the standard Cross-Entropy loss is actually a "coupled" version of HUG, explaining why CE works well but also why it might be sub-optimal compared to the explicit decoupling HUG provides.

Key Results

The paper evaluates HUG across a variety of benchmarks, including Vision (MNIST, CIFAR, ImageNet) and NLP (MRPC, SST-2, WNLI).

  • Superior Accuracy: On CIFAR-10, MHE-HUG achieved a 5.03% error rate, outperforming Cross-Entropy's 5.45%. Similar gains were seen on CIFAR-100 and ImageNet.
  • Adversarial Robustness: This was a standout result. Under a PGD adversarial attack ($l_\infty = 2/255$), a ResNet-18 trained with HUG maintained 15.24% accuracy, while the CE-trained model plummeted to 7.94%.
  • Long-Tailed & Continual Learning: HUG consistently outperformed CE in scenarios where data is imbalanced or arrives in stages, likely because the explicit inter-class separation prevents majority classes from "overwhelming" the feature space.

Limitations and Future Work

  • Hyperparameter Sensitivity: HUG introduces $\alpha$ and $\beta$ weights. While the authors show the model isn't hyper-sensitive, finding the optimal balance for a new task requires more tuning than the "plug-and-play" Cross-Entropy.
  • Implicit Data Mining: Standard CE has an "implicit data mining" property where it focuses more on hard samples near the decision boundary; the current HUG formulation lacks a direct equivalent of this.
  • Feature Norms: The current theory assumes features are projected onto a sphere (ignoring their magnitude/norm). The authors acknowledge that feature norms carry important information (like uncertainty) and suggest incorporating them in future versions of HUG.

Summary updated

Abstract

The neural collapse (NC) phenomenon describes an underlying geometric symmetry for deep neural networks, where both deeply learned features and classifiers converge to a simplex equiangular tight frame. It has been shown that both cross-entropy loss and mean square error can provably lead to NC. We remove NC's key assumption on the feature dimension and the number of classes, and then present a generalized neural collapse (GNC) hypothesis that effectively subsumes the original NC. Inspired by how NC characterizes the training target of neural networks, we decouple GNC into two objectives: minimal intra-class variability and maximal inter-class separability. We then use hyperspherical uniformity (which characterizes the degree of uniformity on the unit hypersphere) as a unified framework to quantify these two objectives. Finally, we propose a general objective -- hyperspherical uniformity gap (HUG), which is defined by the difference between inter-class and intra-class hyperspherical uniformity. HUG not only provably converges to GNC, but also decouples GNC into two separate objectives. Unlike cross-entropy loss that couples intra-class compactness and inter-class separability, HUG enjoys more flexibility and serves as a good alternative loss function. Empirical results show that HUG works well in terms of generalization and robustness.
BibTeX
@inproceedings{liu2023generalizing,
  author = {Weiyang Liu and Longhui Yu and Adrian Weller and Bernhard Schölkopf},
  title = {{Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=inU2quhGdNU},
}
Figure