In modern deep learning, the standard recipe for training a classifier involves mapping high-dimensional inputs into a feature space where samples from the same category cluster tightly together. While this strategy excels at minimizing training error, it often leads to a phenomenon known as representation collapse. As Zhang et al. observe, standard classification objectives frequently force all samples of a category toward a single point in the feature space, effectively destroying the internal structure and nuanced relationships within the data.
The Geometry of Neural Collapse
This convergence is not just a side effect but a mathematical tendency of deep networks. Xu et al. describe this as "Neural Collapse," where features of the same class converge toward a single prototype. To measure this, the authors introduced the Variability Collapse Index (VCI), demonstrating that while excessive collapse might improve training accuracy, it severely harms the model's ability to transfer its knowledge to new, unseen tasks.
From an information-theoretic perspective, Yu et al. argue that traditional supervised losses discard the rich structural information present within individual categories. By mapping diverse samples to a low-dimensional line or a single point, the model loses the "coding rate" (or volume) of the data. This loss of entropy is not limited to classification; Zhang et al. found that standard regression losses like Mean Squared Error also lead to collapsed feature spaces, limiting a model's ability to distinguish fine-grained differences in continuous values like age or depth.
The Cost of Uniformity
The primary victim of representation collapse is fine-grained discrimination. When a model is trained on coarse labels (e.g., "Dog"), the optimization process encourages it to ignore subtle variations to better group diverse samples under that single label. Yang et al. and Min et al. highlight a "granularity gap": a model that treats all dogs as identical in feature space will fail when asked to distinguish between a Beagle and a Collie at test time.
This collapse also creates a fragility in model robustness. Chen et al. note that Supervised Contrastive Learning (SupCon) is particularly prone to "class collapse," which discards the intra-class variations essential for identifying rare subgroups or adapting to distribution shifts. Similarly, Zhou et al. discovered that even Label Smoothing—a technique intended to regularize models—contains an "error-amplification" term that can compress features into excessively tight clusters, hindering the model's effectiveness as a general-purpose feature extractor.
Strategies for Maintaining Diversity
To combat this, researchers have proposed moving away from "point-based" representations toward more expressive geometric structures:
- Subspace Representations: Instead of mapping a class to a single vector, Wang et al. propose Grassmann Class Representation (GCR), which maps classes to high-dimensional linear subspaces. This allows the model to preserve intra-class variance, breaking the traditional trade-off between accuracy and transferability.
- Multi-Prototype Clustering: Rather than forcing one cluster per class, Feng et al. introduced LOOK, a method that allows classes to form multiple distinct clusters. This preserves visual nuances like different viewpoints or lighting conditions that are otherwise discarded.
- Balancing Objectives: Chen et al. developed THANOS, which combines supervised contrastive learning with a class-conditional autoencoder. This forces the model to "spread out" representations while still maintaining class boundaries.
- Regularization Tweaks: To fix the collapse caused by label smoothing, Zhou et al. proposed Max Suppression (MaxSup), which penalizes the model's top prediction instead of the ground truth, effectively restoring feature diversity.
Why Diversity Matters for Generalization
The intuition behind these methods is that the "noise" or "variance" within a class is often actually "dark knowledge"—latent features that are irrelevant for the current task but vital for the next one. As Confavreux et al. point out in the context of language models, supervised fine-tuning often "collapses" internal representations to focus strictly on task labels, leading to high confidence but fragile generalization. By contrast, maintaining a diverse, high-entropy feature space ensures that the model retains the building blocks—textures, sub-components, and structural relationships—needed to adapt to the complex, multi-granular reality of the open world.
Key Foundational Work
How should we evaluate supervised hashing? (2017) — contributed 1 concept elements to this direction.
Embedding Label Structures for Fine-Grained Feature Representation (2016) — contributed 1 concept elements to this direction.