Abstract
Recent advances in theoretical Deep Learning have introduced geometric properties that occur during training, past the Interpolation Threshold -- where the training error reaches zero. We inquire into the phenomena coined Neural Collapse in the intermediate layers of the networks, and emphasize the innerworkings of Nearest Class-Center Mismatch inside the deepnet. We further show that these processes occur both in vision and language model architectures. Lastly, we propose a Stochastic Variability-Simplification Loss (SVSL) that encourages better geometrical features in intermediate layers, and improves both train metrics and generalization.
Introduction
Deep neural networks are often viewed as "black boxes," but recent theoretical work has begun to illuminate their internal geometry. A key focus has been Neural Collapse (NC), a phenomenon typically observed during the "Terminal Phase of Training" (TPT) after training error reaches zero. In this phase, class-specific activations cluster tightly around their means, and the final classifier simplifies to a nearest class-center decision rule. Traditionally, these properties were thought to emerge only in the penultimate layer. This research investigates whether this geometric simplification occurs throughout the network and how it can be leveraged to improve model performance.
Methods: NCC Mismatch and SVSL
The authors introduce the NCC Mismatch metric to quantify the disagreement between a network's actual final prediction and a hypothetical prediction based on the nearest class mean at a specific intermediate layer. Their observations lead to the "Monotone Ordering Conjecture," which posits that this mismatch decreases monotonically as data flows deeper into the network.
To actively encourage this geometric structure, the researchers propose the Stochastic Variability-Simplification Loss (SVSL). SVSL adds a penalty term to the standard Cross-Entropy (CE) loss, minimizing the distance between intermediate activations and their respective class means. To maintain computational efficiency, these means are calculated stochastically per batch rather than across the entire dataset.
Results and Insights
The study spans multiple modalities, including Computer Vision (MNIST, CIFAR, STL-10) and Natural Language Processing (BERT on GLUE benchmarks).
Universal Geometric Behavior
A significant finding is that both CNNs and Transformers exhibit similar NCC-simplification patterns. This suggests that the geometric "uncrumpling" of data is a fundamental property of deep learning architectures regardless of modality. Furthermore, the initial NCC mismatch in the first layer can serve as a proxy for dataset difficulty; for example, MNIST starts with a much lower mismatch (~0.16) than the more complex CIFAR-10 (~0.6).
Performance Gains
Models trained with SVSL consistently outperformed "vanilla" models across various metrics. On the STL-10 dataset, SVSL improved the best test accuracy from 54.19% to 56.94%. In NLP tasks like CoLA, the Matthew’s Correlation score increased from 53.95 to 55.54. These improvements suggest that enforcing geometric "neatness" in intermediate layers directly correlates with better generalization to unseen data.
Conclusion
This work moves Neural Collapse from a theoretical curiosity of the final layer to a practical tool for the entire network architecture. By introducing SVSL, the authors provide a method to guide geometric simplification throughout the model, leading to more robust results. Future work may focus on automating hyperparameter selection for SVSL and exploring "early exit" strategies enabled by early geometric collapse.
Summary updated