Quantifying the Variability Collapse of Neural Networks

Jing Xu, Haoxiong Liu

Abstract

Recent studies empirically demonstrate the positive relationship between the transferability of neural networks and the within-class variation of the last layer features. The recently discovered Neural Collapse (NC) phenomenon provides a new perspective of understanding such last layer geometry of neural networks. In this paper, we propose a novel metric, named Variability Collapse Index (VCI), to quantify the variability collapse phenomenon in the NC paradigm. The VCI metric is well-motivated and intrinsically related to the linear probing loss on the last layer features. Moreover, it enjoys desired theoretical and empirical properties, including invariance under invertible linear transformations and numerical stability, that distinguishes it from previous metrics. Our experiments verify that VCI is indicative of the variability collapse and the transferability of pretrained neural networks.

Introduction

In deep learning, pretrained models are frequently utilized as feature extractors for downstream tasks. A critical observation in recent years is the phenomenon of Neural Collapse (NC). Specifically, the first stage of this phenomenon, known as NC1 or variability collapse, describes how the features of all images within a single class converge toward their class mean as training progresses.

While this convergence is a hallmark of training, excessive collapse can paradoxically impair a model's ability to generalize to new tasks (transferability). To date, quantifying this effect has been difficult. Existing metrics, such as "Fuzziness" or "Squared Distance," suffer from numerical instability—often requiring the inversion of nearly singular matrices—and a lack of invariance under linear transformations. This paper introduces the Variability Collapse Index (VCI) to address these limitations and provide a reliable measure of feature geometry.

The Variability Collapse Index (VCI)

The core innovation of VCI is its grounding in the mathematics of Linear Probing. The authors derive the metric by proving that the minimum Mean Squared Error (MSE) loss for a linear classifier is directly related to the term $Tr[\Sigma_T^\dagger \Sigma_B]$, where $\Sigma_B$ represents between-class covariance and $\Sigma_T$ represents total covariance.

The resulting formula is defined as:

$$VCI = 1 - \frac{Tr[\Sigma_T^\dagger \Sigma_B]}{rank(\Sigma_B)}$$

Key Advantages

  • Numerical Stability: By utilizing the total covariance matrix ($\Sigma_T$) for the pseudo-inverse operation rather than just the between-class matrix, VCI avoids the "explosive" values common in previous metrics.
  • Invariance: VCI is mathematically invariant under any invertible linear transformation. This is crucial because linear probing—the standard method for evaluating features—is also invariant to these transformations. Previous metrics often changed values when features were simply scaled or rotated, creating a gap between the metric and actual model performance.

Experimental Validation

The researchers validated VCI using standard architectures, including ResNet18, ResNet50, and Vision Transformers (ViT/DeiT), pretrained on CIFAR-10 and ImageNet-1K. To assess transferability, these models were evaluated across 10 diverse downstream datasets, such as Oxford-IIIT Pets, FGVC Aircraft, and Stanford Cars.

Key Insights

  1. Predictive Accuracy: VCI serves as a highly reliable "transferability compass." In experiments, it achieved a Pearson correlation of $r = 0.975$ with Mean Log Odds Gain, a standard measure of transfer performance.
  2. Robustness to Regularization: When regularization was applied to control feature diversity, VCI was the only metric that consistently indicated that higher diversity (lower collapse) led to better transfer. Older metrics like Fuzziness often showed confusing or negative correlations ($r = -0.414$) in the same scenarios.
  3. Collapse Dynamics: The study demonstrates that full collapse is not necessary for minimum training loss. In real-world training, some aspects of feature variation actually increase over time, a nuance that VCI captures more effectively than erratic traditional metrics.

Conclusion and Future Work

VCI provides a theoretically grounded and numerically stable framework for understanding the geometry of neural network features. By bridging the gap between variability collapse and linear probing loss, it offers a predictive tool for evaluating pretrained models.

Future research may extend this analysis to the intermediate layers of deep architectures and explore more direct theoretical links between VCI and Cross-Entropy loss, which is more common in modern training than the MSE loss used for the initial derivation. As foundation models continue to scale, metrics like VCI will be essential for monitoring and optimizing feature representation for downstream success.

Summary updated

Abstract

Recent studies empirically demonstrate the positive relationship between the transferability of neural networks and the within-class variation of the last layer features. The recently discovered Neural Collapse (NC) phenomenon provides a new perspective of understanding such last layer geometry of neural networks. In this paper, we propose a novel metric, named Variability Collapse Index (VCI), to quantify the variability collapse phenomenon in the NC paradigm. The VCI metric is well-motivated and intrinsically related to the linear probing loss on the last layer features. Moreover, it enjoys desired theoretical and empirical properties, including invariance under invertible linear transformations and numerical stability, that distinguishes it from previous metrics. Our experiments verify that VCI is indicative of the variability collapse and the transferability of pretrained neural networks.
BibTeX
@inproceedings{xu2023quantifying,
  author = {Jing Xu and Haoxiong Liu},
  title = {{Quantifying the Variability Collapse of Neural Networks}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=PSeePcY7WR},
}
Figure