Limitations of Neural Collapse for Understanding Generalization in Deep Learning

Like Hui, Mikhail Belkin, Preetum NakkiranarXiv 2022 · OpenReview

Abstract

The recent work of Papyan, Han, & Donoho (2020) presented an intriguing "Neural Collapse" phenomenon, showing a structural property of interpolating classifiers in the late stage of training. This opened a rich area of exploration studying this phenomenon. Our motivation is to study the upper limits of this research program: How far will understanding Neural Collapse take us in understanding deep learning? First, we investigate its role in generalization. We refine the Neural Collapse conjecture into two separate conjectures: collapse on the train set (an optimization property) and collapse on the test distribution (a generalization property). We find that while Neural Collapse often occurs on the train set, it does not occur on the test set. We thus conclude that Neural Collapse is primarily an optimization phenomenon, with as-yet-unclear connections to generalization. Second, we investigate the role of Neural Collapse in feature learning. We show simple, realistic experiments where training longer leads to worse last-layer features, as measured by transfer-performance on a downstream task. This suggests that neural collapse is not always desirable for representation learning, as previously claimed. Finally, we give preliminary evidence of a "cascading collapse" phenomenon, wherein some form of Neural Collapse occurs not only for the last layer, but in earlier layers as well. We hope our work encourages the community to continue the rich line of Neural Collapse research, while also considering its inherent limitations.

Introduction

The phenomenon of "Neural Collapse" (NC) describes a structural convergence in the final layers of deep neural networks during the terminal phase of training. Specifically, after reaching zero training error, the last-layer representations of samples within a class tend to collapse toward their class mean, and these means arrange themselves into a symmetric simplex. While initially hypothesized as a potential key to understanding deep learning's generalization capabilities, this work investigates whether NC is a fundamental property of generalization or merely an artifact of optimization.

The Generalization Gap in Collapse

To rigorously study this phenomenon, we distinguish between Train-Collapse and Test-Collapse. Train-collapse refers to the convergence of training sample representations to their respective class means. Test-collapse is further divided into Weak Test-Collapse (test samples mapping to any of the discrete class points) and Strong Test-Collapse (test samples mapping to the correct class mean).

Our empirical investigation across multiple architectures (ResNet, VGG, DenseNet) and datasets (MNIST, FashionMNIST, CIFAR-10, SVHN, STL-10) reveals a consistent trend: while models exhibit near-perfect collapse on the training set, they fail to do so on the test set.

Feature variance across various datasets and architectures. While training variance (magenta) collapses toward zero, test variance (black) remains high.

As training progresses far beyond the point of zero training error, the training variance continues to plummet toward zero. However, the test variance (both strong and weak) plateaus at a significantly higher level. This divergence suggests that Neural Collapse is primarily an optimization phenomenon—a consequence of how stochastic gradient descent (SGD) fits a finite training set—rather than a property of the underlying data distribution.

Anti-Correlation: When More Data Means Less Collapse

A critical test for any generalization metric is its behavior as the training set size ($N$) increases. Typically, larger datasets lead to better generalization. Surprisingly, we find an anti-correlation between generalization and train-time collapse: models trained on larger datasets actually exhibit less collapse on the training set.

Relationship between training set size (N), train-time collapse, and test variance. Larger training sets lead to better generalization but less collapse.

This finding further decouples NC from generalization performance. If "more collapse" were a driver of better generalization, we would expect to see it increase with larger $N$, but the opposite occurs. On CIFAR-10 with a ResNet18, TrainVariance drops to nearly $10^{-5}$, while StrongTestVariance remains above $10^0$.

Implications for Representation Learning

Beyond generalization, we examined whether Neural Collapse is desirable for feature learning. By pre-training models on "super-class" tasks and evaluating them on downstream fine-tuning tasks, we observed that training longer (and thus inducing more collapse) can actually degrade the quality of learned features. Models with highly collapsed representations performed worse on downstream tasks, likely because the collapse process discards fine-grained information that is not strictly necessary for the training task but vital for transferability.

Conclusion

While Neural Collapse remains a fascinating structural property of deep networks in the interpolatory regime, its role in generalization appears limited. Our results indicate that NC is an optimization quirk that does not extend to the test distribution and may even harm representation quality. We also provide preliminary evidence of "cascading collapse" in earlier layers, suggesting that this optimization behavior is more pervasive than previously thought. We encourage the community to explore these limitations to build a more nuanced understanding of deep learning dynamics.

Summary updated

Abstract

The recent work of Papyan, Han, & Donoho (2020) presented an intriguing "Neural Collapse" phenomenon, showing a structural property of interpolating classifiers in the late stage of training. This opened a rich area of exploration studying this phenomenon. Our motivation is to study the upper limits of this research program: How far will understanding Neural Collapse take us in understanding deep learning? First, we investigate its role in generalization. We refine the Neural Collapse conjecture into two separate conjectures: collapse on the train set (an optimization property) and collapse on the test distribution (a generalization property). We find that while Neural Collapse often occurs on the train set, it does not occur on the test set. We thus conclude that Neural Collapse is primarily an optimization phenomenon, with as-yet-unclear connections to generalization. Second, we investigate the role of Neural Collapse in feature learning. We show simple, realistic experiments where training longer leads to worse last-layer features, as measured by transfer-performance on a downstream task. This suggests that neural collapse is not always desirable for representation learning, as previously claimed. Finally, we give preliminary evidence of a "cascading collapse" phenomenon, wherein some form of Neural Collapse occurs not only for the last layer, but in earlier layers as well. We hope our work encourages the community to continue the rich line of Neural Collapse research, while also considering its inherent limitations.
BibTeX
@misc{hui2022limitations,
  author = {Like Hui and Mikhail Belkin and Preetum Nakkiran},
  title = {{Limitations of Neural Collapse for Understanding Generalization in Deep Learning}},
  year = {2022},
  eprint = {2202.08384},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.08384},
}
Figure