Geometric Symmetry for Long Tailed Learning

This direction is split into other directions: geometric symmetry restoration. Its last version stays readable and citable here.

Version 1 of 137 papers49 conceptsfrozen 2026-10-04

In a perfectly balanced dataset, deep neural networks tend to exhibit a phenomenon called Neural Collapse. As training progresses, the features of each class converge toward their respective means, and these class means arrange themselves into a Simplex Equiangular Tight Frame (ETF)—a geometric structure where every class is maximally and equally separated from every other class.

However, real-world data is rarely balanced. In "long-tailed" distributions, a few majority classes dominate while many minority classes have only a handful of samples. This imbalance shatters the natural symmetry of Neural Collapse. Majority classes expand their territory in the feature space, "crowding out" minority classes into infinitesimally small volumes or causing their representations to collapse entirely.

Recent research suggests that we can fix this by explicitly forcing the model to rediscover this geometric harmony. By regularizing the model's internal representations to follow an ETF structure, we can ensure that even the rarest classes have the "breathing room" they need to be accurately identified.

The Geometry of Crowding

When a model is trained on imbalanced data using standard cross-entropy loss, the gradients are dominated by majority classes. Geometrically, this manifests as a bias in the embedding space. As Guanlin Li and colleagues note in Alleviating the Effect of Data Imbalance on Adversarial Training, majority classes tend to develop larger feature norms and wider decision boundaries, pushing minority classes toward the origin or into each other.

Titir Dutta and Soma Biswas demonstrate in Adaptive Margin Diversity Regularizer for Handling Data Imbalance in Zero-Shot SBIR that this spatial imbalance is the primary reason models fail on rare or unseen categories. Traditional fixes like over-sampling the minority data often fail because they don't address this underlying geometric distortion. Instead, researchers are turning to Angular Regularization, a technique introduced by Jianxin Li in Improving the Generalization Performance of Multi-class SVM via Angular Regularization, which forces the angles between class hyperplanes to be larger, preserving the functional space required for minority class identification.

Forcing Symmetry with Equiangular Tight Frames

The most potent tool for restoring this balance is the Simplex Equiangular Tight Frame (ETF). An ETF is a set of vectors where the angle between any two vectors is identical and as large as possible. In a $d$-dimensional space with $K$ classes, the optimal separation is achieved when the vectors form a simplex.

Rather than letting the model learn its own (likely biased) classifier weights, researchers like Yang Shen propose using Equiangular Basis Vectors. By replacing the final learnable layer with a fixed, non-learnable ETF, the model is forced to map features into a pre-defined, perfectly balanced space. This "fixed target" approach ensures that every category, regardless of its frequency, occupies an equivalent portion of the representation space.

This mechanism has proven effective across diverse domains:

All-around Neural Collapse

A key insight from Enhao Zhang and Songcan Chen in All-around Neural Collapse for Imbalanced Classification is that regularizing only the classifier or only the features is insufficient. They argue for self-duality: the state where class feature means and classifier weights are perfectly aligned to the same ETF.

When this alignment is broken—a state they call "minority collapse"—the model's internal logic becomes inconsistent. By simultaneously aligning both features and weights to a shared geometric target, the AllNC framework restores the structural integrity of the network. This is echoed by Xuantong Liu in Inducing Neural Collapse in Deep Long-tailed Learning, who found that minimizing intra-class variance while maximizing inter-class separation through ETF targets is the most effective way to "induce" collapse in imbalanced settings.

Why Geometric Regularization Works

The intuition behind these methods is that geometry acts as a proxy for missing data. In a long-tailed dataset, we don't have enough samples to define the boundaries of the "tail" classes. By imposing an ETF structure, we are essentially telling the model: "I don't have many examples of this class, but I know it should be exactly as distinct from Class A as Class B is."

This structural prior prevents the "norm bias" described by Ying Wang in Spherical Procrustes Alignment for Reliable Medical Audio Diagnosis. In high-stakes fields like medicine, models often become overconfident in majority classes because their feature vectors are longer (have higher norms). By enforcing norm consistency—making all class vectors the same length—and aligning them to an ETF, we force the model to rely on semantic meaning rather than sample frequency.

As Shenzhi Yang shows in NodeReg, this geometric shift transforms the feature distribution into a "ring-like" structure, creating a wider, more equitable margin that allows rare classes to finally stand on equal footing with the majority.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_geometric_symmetry_for_long_tailed_learn_a5c4c69d,
  title = {Geometric Symmetry for Long Tailed Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometric-symmetry-for-long-tailed-learning/rdr_a5c4c69df00e45f1a9cdc8821c81cbaf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_geometric_symmetry_for_long_tailed_learn_a5c4c69d_v1,
  title = {Geometric Symmetry for Long Tailed Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometric-symmetry-for-long-tailed-learning/rdr_a5c4c69df00e45f1a9cdc8821c81cbaf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}