In a perfectly balanced dataset, deep neural networks tend to exhibit a phenomenon called Neural Collapse. As training progresses, the features of each class converge toward their respective means, and these class means arrange themselves into a Simplex Equiangular Tight Frame (ETF)—a geometric structure where every class is maximally and equally separated from every other class.
However, real-world data is rarely balanced. In "long-tailed" distributions, a few majority classes dominate while many minority classes have only a handful of samples. This imbalance shatters the natural symmetry of Neural Collapse. Majority classes expand their territory in the feature space, "crowding out" minority classes into infinitesimally small volumes or causing their representations to collapse entirely.
Recent research suggests that we can fix this by explicitly forcing the model to rediscover this geometric harmony. By regularizing the model's internal representations to follow an ETF structure, we can ensure that even the rarest classes have the "breathing room" they need to be accurately identified.
The Geometry of Crowding
When a model is trained on imbalanced data using standard cross-entropy loss, the gradients are dominated by majority classes. Geometrically, this manifests as a bias in the embedding space. As Guanlin Li and colleagues note in Alleviating the Effect of Data Imbalance on Adversarial Training, majority classes tend to develop larger feature norms and wider decision boundaries, pushing minority classes toward the origin or into each other.
Titir Dutta and Soma Biswas demonstrate in Adaptive Margin Diversity Regularizer for Handling Data Imbalance in Zero-Shot SBIR that this spatial imbalance is the primary reason models fail on rare or unseen categories. Traditional fixes like over-sampling the minority data often fail because they don't address this underlying geometric distortion. Instead, researchers are turning to Angular Regularization, a technique introduced by Jianxin Li in Improving the Generalization Performance of Multi-class SVM via Angular Regularization, which forces the angles between class hyperplanes to be larger, preserving the functional space required for minority class identification.
Forcing Symmetry with Equiangular Tight Frames
The most potent tool for restoring this balance is the Simplex Equiangular Tight Frame (ETF). An ETF is a set of vectors where the angle between any two vectors is identical and as large as possible. In a $d$-dimensional space with $K$ classes, the optimal separation is achieved when the vectors form a simplex.
Rather than letting the model learn its own (likely biased) classifier weights, researchers like Yang Shen propose using Equiangular Basis Vectors. By replacing the final learnable layer with a fixed, non-learnable ETF, the model is forced to map features into a pre-defined, perfectly balanced space. This "fixed target" approach ensures that every category, regardless of its frequency, occupies an equivalent portion of the representation space.
This mechanism has proven effective across diverse domains:
- Semantic Segmentation: Jiequan Cui and Xiaojuan Qi introduced the Center Collapse Regularizer (CeCo) in Understanding Imbalanced Semantic Segmentation Through Neural Collapse to force feature centers into symmetry, protecting minority class gradients from being overwhelmed.
- Federated Learning: Chenxi Huang developed FedETF in Neural Collapse Inspired Federated Learning with Non-iid Data, which uses a global ETF to prevent local models from drifting toward majority classes in decentralized settings.
- Large Language Models: Hanglei Hu applied these principles to LLMs in Advancing Personalized Learning with Neural Collapse for Long-Tail Challenge, showing that forcing text embeddings into an ETF structure improves generalization on rare educational data.
All-around Neural Collapse
A key insight from Enhao Zhang and Songcan Chen in All-around Neural Collapse for Imbalanced Classification is that regularizing only the classifier or only the features is insufficient. They argue for self-duality: the state where class feature means and classifier weights are perfectly aligned to the same ETF.
When this alignment is broken—a state they call "minority collapse"—the model's internal logic becomes inconsistent. By simultaneously aligning both features and weights to a shared geometric target, the AllNC framework restores the structural integrity of the network. This is echoed by Xuantong Liu in Inducing Neural Collapse in Deep Long-tailed Learning, who found that minimizing intra-class variance while maximizing inter-class separation through ETF targets is the most effective way to "induce" collapse in imbalanced settings.
Why Geometric Regularization Works
The intuition behind these methods is that geometry acts as a proxy for missing data. In a long-tailed dataset, we don't have enough samples to define the boundaries of the "tail" classes. By imposing an ETF structure, we are essentially telling the model: "I don't have many examples of this class, but I know it should be exactly as distinct from Class A as Class B is."
This structural prior prevents the "norm bias" described by Ying Wang in Spherical Procrustes Alignment for Reliable Medical Audio Diagnosis. In high-stakes fields like medicine, models often become overconfident in majority classes because their feature vectors are longer (have higher norms). By enforcing norm consistency—making all class vectors the same length—and aligning them to an ETF, we force the model to rely on semantic meaning rather than sample frequency.
As Shenzhi Yang shows in NodeReg, this geometric shift transforms the feature distribution into a "ring-like" structure, creating a wider, more equitable margin that allows rare classes to finally stand on equal footing with the majority.