Abstract
This paper formalizes "feature collapse"—the phenomenon where models assign identical representations to different entities that serve the same functional role in a task. Through a prototypical NLP task, the authors prove that normalization layers like LayerNorm are the essential mechanism that enables this collapse in long-tailed data, directly driving the model's ability to generalize.
Introduction
In deep learning, "good features" are often described as representations that discard noise while preserving task-relevant information. For instance, a model should ideally treat various instances of "grass" as a single concept, regardless of pixel-level variance. Laurent, von Brecht, and Bresson formalize this intuition as feature collapse, where a model maps different entities with the same functional role to identical representations.
A significant hurdle to this collapse is the frequency trap. In real-world datasets, which typically follow a Zipfian distribution, frequent items often receive larger embedding magnitudes than rare ones. This imbalance prevents the model from grouping them into a single concept, leading to poor generalization on the long tail.
Methodology: The Synthetic Data Model
To study this phenomenon rigorously, the authors developed a synthetic NLP classification task. The vocabulary consists of words partitioned into discrete "concepts" (e.g., veggie, dairy, meat). Sentences are generated by latent variables that select concepts, which then sample specific words based on predefined distributions.
The researchers compared two linear architectures: a standard embedding-and-concatenate model and one incorporating a LayerNorm module before concatenation. The study evaluated these models on both uniform distributions (equal word frequency) and long-tailed distributions (Zipfian).
The Role of Normalization
The core insight of the paper is that LayerNorm acts as a "symmetry restorer." In long-tailed data, gradient descent naturally pushes frequent words toward larger embedding magnitudes. This "unbalanced" state prevents the model from collapsing rare and frequent words into the same concept.
LayerNorm's primary function in this context is to strip away frequency information. By forcing all embeddings to have the same magnitude, it removes the "noise" of occurrence frequency, allowing the model to focus exclusively on the "direction" (the concept). This restoration of symmetry is what enables perfect feature collapse.
Results and Theoretical Proof
The authors' findings highlight a stark contrast between normalized and unnormalized models:
- Uniform Case: When all words appear equally often, both standard and LayerNorm models achieve 100% accuracy and show perfect feature collapse.
- Long-tailed Case: Without normalization, accuracy dropped to 45%. The standard model achieved "directional collapse" (words pointing in the correct direction) but failed at "magnitude collapse."
- Normalization Success: The LayerNorm model maintained 100% accuracy on long-tailed data by achieving full feature collapse.
The paper provides mathematical proofs (Theorems 1-3) demonstrating that under specific symmetry assumptions, the global minimizer of the risk function must be a state of perfect feature collapse. This is defined using Equiangular Tight Frames (ETFs), configurations where vectors are maximally spread out.
Conclusion
"Feature collapse" provides a rigorous framework for understanding how models learn task-dependent concepts. While "Neural Collapse" describes the behavior of the final layer, feature collapse occurs at the input level and is driven by architectural choices like LayerNorm. Future work will need to explore how these findings translate to non-linear architectures and more complex, less symmetric real-world datasets.
Summary updated