When deep neural networks are trained on heavily imbalanced or "long-tailed" datasets, majority classes dominate the optimization process. From a geometric perspective, this gradient bias distorts the network's final embedding space. The majority classes expand their territory, pushing decision boundaries outward, while minority classes are compressed into narrow regions or completely subsumed.
This distortion breaks a phenomenon known as Neural Collapse ($\mathcal{NC}$)—the natural tendency of deep networks to organize class features into perfectly symmetric geometric structures at the end of training. As shown by Enhao Zhang et al. in All-around Neural Collapse for Imbalanced Classification, class imbalance causes "minority collapse," where both classifier weights and feature means for rare classes lose their distinctiveness and drift toward dominant categories.
To combat this, a powerful paradigm has emerged: geometric symmetry restoration. Instead of allowing the network to learn its final classifier layer from scratch, researchers impose rigid, mathematically optimal geometric targets that force the network to treat all categories equally.
The Simplex ETF as a Static Shield
The cornerstone of geometric symmetry restoration is the Simplex Equiangular Tight Frame (ETF). A Simplex ETF is a geometric structure where a set of vectors is distributed across a hypersphere such that every single pair shares the exact same maximum angular separation.
Standard Training (Imbalanced) Geometric Symmetry Restoration
Majority Class (Wide) Fixed Simplex ETF Target
/ | \ / | \
/ | \ / | \
* * * * * *
/ \ / | \
Minority (Collapsed) Minority Majority Minority
(Equal Representational Capacity)
In standard training, the classifier weights adapt to the data distribution, allowing head classes to crowd out tail classes. To prevent this, Yibo Yang et al. proposed a radical shift in Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?: fix the final linear layer as a static Simplex ETF. Because these class prototypes are frozen and cannot be updated, they act as an unyielding geometric shield. The majority classes cannot encroach on the representational territory of minority classes, forcing the feature extractor to discover balanced representations despite the skewed data.
To add flexibility to this rigid constraint, Gao Peifeng et al. introduced a framework in Feature Directions Matter: Long-Tailed Learning via Rotated Balanced Representation where the ETF is allowed to rotate as a single rigid body during training. This preserves the perfect equiangular separation between classes while allowing the model to align the entire geometric structure with the most informative feature directions. Similarly, in the context of contrastive learning, Jianggang Zhu et al. developed Balanced Contrastive Learning for Long-Tailed Visual Recognition, which enforces a regular simplex geometry on the hypersphere so that every class embedding remains equidistant from all others, reserving equal representational capacity for rare categories.
Aligning Features and Inducing Collapse
Fixing the classifier layer is only half the solution; the network's internal feature representations must also be actively guided to collapse into these symmetric targets.
Xuantong Liu et al. demonstrated in Inducing Neural Collapse in Deep Long-tailed Learning that adding explicit regularization terms to minimize intra-class variance and maximize inter-class separation toward an ETF configuration successfully restores symmetry. This optimization strategy ensures that minority class centers do not drift toward majority classes.
A key refinement here is the concept of self-duality—the state where class feature means and classifier weights perfectly align with the same geometric target. The authors of All-around Neural Collapse for Imbalanced Classification found that regularizing only the classifier creates a geometric mismatch. By simultaneously aligning both the feature means and the classifier weights to a shared Simplex ETF, they maintain structural harmony across the network.
For complex spatial tasks like semantic segmentation, where spatial context can worsen class imbalance, Jiequan Cui and Zhisheng Zhong introduced a center collapse regularizer in Understanding Imbalanced Semantic Segmentation Through Neural Collapse. This mechanism maps the mean feature centers of each class to a fixed ETF, protecting the gradients of rare object categories from being overwhelmed by dominant background classes.
Shifting to Orthogonal Geometries
While the Simplex ETF is the ideal geometry for unconstrained feature spaces, common architectural choices—such as non-negativity constraints—fundamentally alter the optimal geometric landscape.
Ganesh Ramachandra Kini et al. discovered in Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching that simply applying a ReLU activation to the final embedding layer shifts the optimal geometric arrangement from a simplex to an orthogonal frame. In this configuration, class means are anchored at stable 90-degree angles relative to one another. Remarkably, this orthogonal structure remains geometrically stable regardless of how lopsided the training data distribution is.
Simplex ETF Geometry Orthogonal Frame Geometry (with ReLU)
(Equal angles, >90°) (Stable 90° angles)
Class 1 Class 2
/ \ |
/ \ |
Class 2 --- Class 3 Class 1 --- Origin
Further theoretical analysis by Hien Dang et al. in Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Features Model revealed that under severe class imbalance, unconstrained ReLU features naturally form an orthogonal structure where the angles remain perfectly orthogonal to preserve separability, but the lengths of the class vectors vary proportionally with class frequency. This mathematical blueprint allows researchers to precisely calibrate loss hyperparameters to the data imbalance ratio, recovering structural symmetry by balancing vector magnitudes.
Why It Works: Neutralizing Gradient Bias
The core intuition behind geometric symmetry restoration is the complete decoupling of the representation space from sample frequencies.
In standard empirical risk minimization, the loss function incentivizes the network to prioritize majority classes because they contribute the vast majority of the gradients. By pre-defining equidistant class centers—whether through fixed prototypes on opposite poles for binary problems (David Mildenberger et al. in A Tale of Two Classes: Adapting Supervised Contrastive Learning to Binary Imbalanced Datasets), uniform hyperspherical distributions (Siming Fu et al. in Uniformly Distributed Category Prototype-Guided Vision-Language Framework for Long-Tail Recognition), or dynamic prototype assignments (Mohammad Saeed Ebrahimi Saadabadi et al. in Hyperspherical Classification with Dynamic Label-to-Prototype Assignment)—the model is provided with a constant, unyielding signal for class separation.
This geometric constraint acts as an architectural counterweight to gradient bias. It ensures that the network reserves distinct, high-quality, and equal-volume territories in the embedding space for every single category, guaranteeing that rare "tail" classes remain just as discriminable as the most frequent "head" classes.
Key Foundational Work
Improving the Generalization Performance of Multi-class SVM via Angular Regularization (2017) — contributed 1 concept elements to this direction.
Gaussian Affinity for Max-Margin Class Imbalanced Learning (2019) — contributed 1 concept elements to this direction.