Abstract
Neural Collapse (NC) describes a state in deep learning where intra-class features converge to their means, leading to maximum class separation. While significant, analyzing NC via traditional Shannon entropy is computationally prohibitive in high-dimensional spaces. This paper introduces matrix entropy as a robust alternative. By proposing two new metrics, the Matrix Mutual Information Ratio (MIR) and Matrix Entropy Difference Ratio (HDR), we provide tools to monitor training dynamics and develop a novel Cross-Model Alignment (CMA) loss. Our results demonstrate that these metrics effectively track NC and significantly improve performance in few-shot and semi-supervised learning tasks.
Introduction
In the terminal phase of training, deep neural networks often exhibit "Neural Collapse" (NC). In this state, intra-class features become nearly identical, and class means align with classifier weights, simplifying the model's geometric structure. Despite its importance, tracking NC is challenging. Traditional probability-based entropy requires estimating distributions in high-dimensional feature spaces, which is often computationally impractical. Furthermore, there is a lack of robust metrics to quantify the information interplay between data representations and classification heads, which is vital for understanding phenomena like "grokking."
Methodology: Matrix Information Theory
To overcome the limitations of Shannon entropy, we utilize Matrix Information Theory. Instead of probability distributions, we compute entropy based on the eigenvalues of a normalized Gram matrix, which represents feature similarities.
Core Metrics
- Matrix Entropy: Captures the information content of the feature space directly from the eigenvalues $\lambda_i$ of the Gram matrix.
- Matrix Mutual Information Ratio (MIR): Measures the shared information between features and weights.
- Matrix Entropy Difference Ratio (HDR): Measures the discrepancy in information levels between features and weights.
Theoretically, as a model approaches Neural Collapse, HDR should converge to 0, and MIR should approach 1.
Cross-Model Alignment (CMA) Loss
We propose the CMA loss to minimize intra-class matrix entropy across different modalities (e.g., image and text). Unlike global entropy reduction methods that risk "class collapse," CMA encourages tighter clustering within classes while maintaining inter-class separation.
Experimental Results
The proposed metrics and loss function were evaluated across standard benchmarks (CIFAR-10/100), few-shot learning (11 datasets), and semi-supervised learning (STL-10).
Neural Collapse Verification
Experimental observations confirm our theoretical derivations. As training progresses and the model reaches the NC state, MIR and HDR values stabilize toward their predicted limits.
Performance Gains
- Few-Shot Learning: CMA consistently outperformed baseline and other entropy-based methods (like VNE) across diverse datasets using the CoOp framework.
- Semi-Supervised Learning: Integrating our matrix-based loss into the FreeMatch algorithm significantly reduced error rates. On CIFAR-10 with 10 labels, the error rate dropped from 8% to 4.69%.
- Grokking Signatures: MIR and HDR act as early indicators for grokking. A local minimum in MIR and a local maximum in HDR typically precede the sudden jump in generalization performance.
Conclusion and Future Work
Matrix entropy provides a computationally efficient and theoretically grounded framework for analyzing Neural Collapse. While eigenvalue decomposition poses some scaling challenges for extremely large batches, the proposed MIR and HDR metrics offer deep insights into model training dynamics. Future work will focus on extending the theoretical framework beyond MSE loss and developing adaptive weighting schemes for the CMA loss.
Summary updated