When deep neural networks are trained to optimality on classification tasks, their final-layer features undergo a fascinating geometric simplification known as Neural Collapse. In this state, the features of each class contract into a single compact representation, and these class centers spread out across the embedding space as far apart as possible. Mathematically, they form a Simplex Equiangular Tight Frame (ETF)—a structure where every single class pair shares the exact same maximal angular separation.
However, this geometric perfection relies on a hidden assumption: the embedding space must have enough room. To place $C$ points equidistantly on a hypersphere, the embedding dimension $d$ must be at least $C - 1$.
When the number of classes outpaces the available dimensions ($C > d$), the embedding space hits a hard ceiling. This phenomenon—geometric saturation—fundamentally alters the optimization landscape and limits the model's capacity to separate features.
The Geometry of the Bottleneck
To build intuition for why this bottleneck occurs, consider a simple low-dimensional analogy. If you want to place three points on a sheet of paper (a 2D space) so that every point is equidistant from the others, you can easily draw an equilateral triangle. If you want to add a fourth equidistant point, you can no longer do it on the flat paper; you must step into the third dimension to build a regular tetrahedron.
Every time you add a class, you require an additional orthogonal direction to maintain perfect symmetry. As noted by the authors of On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features (including first author Jinxin Zhou), if the dimensionality is insufficient, the network cannot reach this ideal state and instead converges to a lower-rank geometric approximation.
When $d < C - 1$, the consequences are severe:
- Convex Hull Entrapment: In high-cardinality tasks, squeezing too many categories into a low-dimensional space forces some class centers to become trapped within the convex hull formed by others. As Zhang et al. (Qizhi Zhang) demonstrate in Large scale classification in deep neural network with Label Mapping, this entrapment destroys linear separability and leads to performance collapse.
- Loss of Equiangularity: The network can no longer maintain equidistant class representations. In extreme multi-label learning environments, Li et al. (Pengyu Li) show in Neural Collapse in Multi-label Learning with Pick-all-label Loss that this compressed regime creates a structural bottleneck that prevents the realization of ideal geometric arrangements.
Cascading Failures Across Domains
The geometric constraints of embedding spaces create practical bottlenecks across several deep learning paradigms.
Continual and Class-Incremental Learning
A popular strategy to mitigate catastrophic forgetting is to pre-allocate a fixed geometric classifier (like an ETF) from the start, forcing the network to align new features to a stable target. This strategy is explored by Li et al. (Zexi Li) in No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier.
However, this approach hits a hard wall when scaling. Seo et al. (Minhyuk Seo) point out in Learning Equi-Angular Representations for Online Continual Learning that if the number of concepts encountered across sequential tasks exceeds the embedding dimension, the equiangular structure collapses. The reduction in angular distance between class prototypes in a crowded feature space eventually degrades the model's ability to distinguish between similar classes. Wang et al. (Zheng Wang) in Rethinking Continual Learning with Progressive Neural Collapse and Baecchi et al. (Claudio Baecchi) in Class-incremental Learning with Pre-allocated Fixed Classifiers confirm that these mathematical properties impose a strict scalability ceiling on continual learning systems.
Out-of-Distribution Detection
Detecting anomalous data often relies on the availability of "empty" space. In Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detection, Wu et al. (Yingwen Wu) show that effective out-of-distribution (OOD) detection works by isolating anomalous data in redundant, unused feature dimensions. When the number of known classes approaches or exceeds the total feature dimensionality, this redundant orthogonal space vanishes, severely limiting the model's ability to distinguish between known and unknown data.
Deep Hashing and Retrieval
In large-scale retrieval, high-dimensional features are compressed into compact binary hash codes. When the number of target categories significantly outnumbers the available bits (dimensions), the space becomes saturated. As Hoe et al. (Jiun Tian Hoe) demonstrate in One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective, and as shown in the single-author paper Learning Attribute-Aware Hash Codes for Fine-Grained Image Retrieval via Query Optimization by Wang (Peng Wang), this mismatch forces a rise in the theoretical minimum similarity between distinct classes, making it inherently difficult to maintain class separability in compact representation spaces.
Bypassing the Dimensionality Limit
To operate effectively in the compressed regime ($C > d$), methods must be designed to function without relying on standard linear separation or strict equiangularity.
One prominent direction is Generalized Neural Collapse. In Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap, Liu et al. (Weiyang Liu) explicitly model the behavior of feature collapse when the feature dimension is smaller than the number of classes. They introduce a loss function that decouples intra-class compactness from inter-class separability, allowing the model to maintain generalization and adversarial robustness even when the ideal simplex structure is mathematically unavailable.
Another approach rethinks how labels are mapped to the geometry of the sphere. In the single-author work Hyperspherical Classification with Dynamic Label-to-Prototype Assignment, Saadabadi (Mohammad Saeed Ebrahimi Saadabadi) proposes replacing learnable softmax layers with fixed, equidistributed prototypes. By dynamically re-assigning class labels to these prototypes during training via bipartite matching, the model optimizes metric space utilization, achieving superior performance even when the feature dimensionality is significantly lower than the number of classes.
Key Foundational Work
Optimization of symbolic feature extraction for pattern classification (2012) — contributed 1 concept elements to this direction.
Guest Editor's Introduction (2003) — contributed 1 concept elements to this direction.
Recognising trajectories of facial identities using kernel discriminant analysis (2003) — contributed 1 concept elements to this direction.