Nearest Class-Center Simplification through Intermediate Layers

Ido Ben-Shaul, Shai Dekel

Abstract

Recent advances in theoretical Deep Learning have introduced geometric properties that occur during training, past the Interpolation Threshold -- where the training error reaches zero. We inquire into the phenomena coined Neural Collapse in the intermediate layers of the networks, and emphasize the innerworkings of Nearest Class-Center Mismatch inside the deepnet. We further show that these processes occur both in vision and language model architectures. Lastly, we propose a Stochastic Variability-Simplification Loss (SVSL) that encourages better geometrical features in intermediate layers, and improves both train metrics and generalization.

Introduction

In the terminal phase of training (TPT)—the period after a neural network has achieved zero training error—a phenomenon known as "Neural Collapse" (NC) typically emerges. In this state, activations in the penultimate layer cluster tightly around their respective class means, effectively turning the final classifier into a simple Nearest Class-Center (NCC) decision maker. While this behavior is well-documented in the final layers, the geometric evolution within the network's intermediate layers has remained less understood. This paper investigates whether the network "untangles" data gradually across its depth and proposes a method to leverage this geometric simplification to improve model performance.

Methodology: Stochastic Variability-Simplification Loss (SVSL)

To investigate and encourage geometric simplification, we introduce the Stochastic Variability-Simplification Loss (SVSL). This loss function augments the standard Cross-Entropy objective with a penalty term designed to minimize the distance between intermediate activations and their class means.

To maintain computational efficiency, SVSL employs stochastic class-mean estimation, calculating means per batch rather than across the entire dataset. The implementation utilizes two primary hyperparameters: $\alpha$, which controls the strength of the simplification penalty, and $\gamma$, which specifies the "starting layer" for the loss. The latter is crucial, as early layers (e.g., those processing raw pixels) may lack the representational capacity to effectively cluster complex data into class centers.

Geometric Evolution in Intermediate Layers

Our analysis reveals a clear layer-wise hierarchy in geometric simplification. By measuring "NCC Mismatch"—the frequency with which an intermediate layer's nearest class-center prediction disagrees with the final network output—we observe that mismatch decreases monotonically with depth.

Evolution of NCC Mismatch across layers for MNIST, Fashion-MNIST, and CIFAR10, showing the monotonic decrease in mismatch as features move deeper into the network.

This evolution is remarkably universal. While demonstrated on ResNet architectures for vision tasks (MNIST, CIFAR10, STL-10), we found that Transformer architectures in NLP (BERT) exhibit the same geometric progression. Furthermore, the NCC mismatch in the initial layers serves as a proxy for dataset difficulty; for instance, MNIST begins with a mismatch of approximately 0.16, whereas the more complex CIFAR10 starts near 0.6. These findings provide empirical support for the "Folding Ball Hypothesis," suggesting that deep networks function by progressively "uncrumpling" data manifolds into linearly separable classes.

Experimental Results and Generalization

The application of SVSL consistently enhances generalization across a variety of benchmarks. In vision tasks, SVSL-trained models outperformed vanilla baselines, notably increasing STL-10 test accuracy from 54.19% to 56.94%. Similar gains were observed in NLP tasks; on the CoLA dataset, the Matthew’s Correlation metric improved from 53.95 to 55.54.

Comparative performance of Vanilla and SVSL models across various vision and NLP datasets, highlighting improvements in both Interpolation Threshold (IT) and End of Training (EOT) metrics.

Beyond raw accuracy, SVSL accelerates the convergence toward Neural Collapse during the TPT, resulting in more stable and geometrically structured feature representations. In nearly every tested scenario, the peak test performance occurred during the TPT, with SVSL reaching higher maximums than standard training.

Conclusion and Future Work

This work demonstrates that Neural Collapse is a network-wide process that can be explicitly optimized. By encouraging intermediate layers to simplify their representations via SVSL, we achieve superior generalization in both vision and language models.

Future research will focus on automating the selection of SVSL hyperparameters ($\alpha$ and $\gamma$) through learnable weights. Additionally, the high degree of NCC-Collapse in intermediate layers suggests the potential for "early-exit" inference strategies, where predictions are made at intermediate stages to reduce computational overhead. Finally, testing these principles on massive foundation models remains a key area for further exploration.

Summary updated

Abstract

Recent advances in theoretical Deep Learning have introduced geometric properties that occur during training, past the Interpolation Threshold -- where the training error reaches zero. We inquire into the phenomena coined Neural Collapse in the intermediate layers of the networks, and emphasize the innerworkings of Nearest Class-Center Mismatch inside the deepnet. We further show that these processes occur both in vision and language model architectures. Lastly, we propose a Stochastic Variability-Simplification Loss (SVSL) that encourages better geometrical features in intermediate layers, and improves both train metrics and generalization.
BibTeX
@misc{benshaul2022nearest,
  author = {Ido Ben-Shaul and Shai Dekel},
  title = {{Nearest Class-Center Simplification through Intermediate Layers}},
  year = {2022},
  eprint = {2201.08924},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.08924},
}
Figure