When training a deep neural network for classification, a common milestone is reaching zero training error. Intuitively, one might expect the network's internal representations to stop changing significantly once all training samples are correctly classified. However, continuing training past this point—into what is known as the "terminal phase of deep learning"—reveals a striking and pervasive phenomenon: the network's internal representations simplify into an elegant, highly symmetric geometric structure.
This phenomenon, first identified by Papyan et al., is called Neural Collapse. It reveals that over-parameterized networks naturally discard individual sample details in favor of unified class prototypes once the training error is minimized.
The Four Facets of Collapse
To understand why and how neural collapse occurs, it helps to look at how the geometry of the final hidden layer changes during extended training. Neural collapse is typically characterized by a synchronized transition across features, class means, and classifier weights:
- Variability Collapse: The internal representations (embeddings) of all inputs belonging to a specific class contract sharply toward their respective class mean. As training progresses in the terminal phase, the within-class variance vanishes, meaning the network begins to treat different instances of the same class as nearly identical points in the feature space.
- Convergence to a Simplex ETF: The class means do not just cluster randomly; they arrange themselves into a highly symmetric geometric structure called a Simplex Equiangular Tight Frame (ETF). In this state, the vectors representing the class centers are maximally and equally separated from one another on a high-dimensional hypersphere.
- Classifier Alignment: The weights of the final linear classification layer choose to mirror these class means perfectly. The final layer's weight vectors align with the collapsed feature clusters, a state of information-theoretic mirroring analyzed by Song et al..
- Simplification of the Decision Rule: Because the features have collapsed to single points and the classifier weights match them perfectly, the network's complex decision boundary simplifies into a nearest-class-mean classification rule.
Why Does It Happen?
Neural collapse is not a quirk of a specific loss function or architecture; it is an inherent property of the optimization landscape of over-parameterized models.
Universality Across Losses
Initially observed under standard Cross-Entropy loss, researchers quickly discovered that this geometric collapse is a universal destination. You et al. demonstrated that diverse classification objectives—including Cross-Entropy, Focal Loss, and Mean Squared Error (MSE)—all converge to the exact same neural collapse geometry when trained to completion. Similarly, Zhou et al. and Hu et al. proved that square and MSE losses naturally possess a benign optimization landscape where global minima correspond to a state of neural collapse.
The Role of the Loss Landscape and Normalization
As Wu demonstrates, this vanishing within-class variability is a fundamental property of the loss landscape itself: any multi-layer architecture reaching a near-stationary point with low empirical loss will naturally favor collapsed representations.
But what drives the specific, equidistant symmetry of the Simplex ETF? The answer lies heavily in architectural choices like normalization layers. Ergen et al. showed via convex duality that data whitening and batch normalization actively drive networks toward low-dimensional, structured configurations. Because normalization layers bound the scale of the representations, the optimization process satisfies its drive for maximum margin separation by spreading the class vectors as far apart as possible. As explained by Kunin et al., neural collapse is the predictable outcome of this maximum margin bias in networks with normalization.
Universality Beyond Standard Classification
The drive toward a collapsed geometric state extends far beyond standard supervised image classification.
- Contrastive Learning: In both supervised and unsupervised contrastive learning, objectives are designed to pull positive pairs together and push negative pairs apart. Dikkala et al. and Thrampoulidis et al. proved that at their global minima, contrastive losses force representations to collapse into equidistant vertices on a hypersphere, minimizing intra-class variance while maximizing inter-class margins.
- Language Models: Even in causal large language models (LLMs), which violate many of the theoretical assumptions of standard classification setup, a similar geometric simplification occurs. Wu et al. observed "linguistic collapse," where internal representations of the same token across different contexts contract toward a single mean vector as the models scale.
- Grokking and Compression: This late-phase geometric contraction also provides a unified explanation for other deep learning mysteries. Sakamoto et al. showed that the slow, late-stage emergence of neural collapse is the underlying mechanism driving both grokking (sudden generalization long after overfitting) and the information bottleneck (the compression of irrelevant input data).
Exploiting the Geometry: Inducing Collapse
If neural collapse represents an ideal, maximally separated feature space, do we have to wait for the terminal phase of training for it to emerge organically? More importantly, what happens when the data distribution is highly imbalanced, disrupting this natural symmetry?
When datasets have long-tailed distributions, the emergent neural collapse can become distorted, causing minority classes to collapse poorly or lose their separation. To counteract this, researchers have shifted from treating neural collapse as a passive byproduct to using it as an explicit structural constraint.
Standard Training:
Data + Learnable Classifier ──(Extended Training)──> Emergent Neural Collapse (ETF)
Induced Collapse:
Data + Fixed ETF Classifier ────────────────────────> Forced Optimal Feature Separation
Yang et al. challenged the standard practice of training the final linear layer by fixing the classifier's weights to a predefined, mathematically perfect Simplex ETF from the start. By removing the classifier's ability to adapt to class frequencies, the network is forced to map its features to this rigid "geometric North Star," inducing optimal feature separation even on highly imbalanced datasets.
This strategy of fixing the final layer to a geometric frame has proven highly effective across several domains:
- Federated Learning: Li et al. utilized fixed ETF classifiers to eliminate classifier bias caused by heterogeneous local data distributions across clients.
- Open-World Learning: Wang et al. applied fixed geometric targets to prevent representation collapse and maintain high class distinction when identifying novel, unseen categories.
- Domain Adaptation: Yu et al. leveraged predefined neural collapse states to preserve class boundaries when adapting models to new target domains without labels.
By understanding neural collapse, we see that the final stages of deep learning optimization operate as a geometric architect—stripping away individual sample variances to build a perfectly symmetrical, maximally separated space for classification.
Key Foundational Work
Prevalence of Neural Collapse during the terminal phase of deep learning training (2020) — contributed 2 concept elements to this direction.
Breaking Inter-Layer Co-Adaptation by Classifier Anonymization (2019) — contributed 2 concept elements to this direction.