Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training

Kun Song, Zhiquan Tan, Bochao Zou, Jiansheng Chen, Huimin Ma, Weiran HuangarXiv 2024 · DOI · arXiv

Abstract

Neural Collapse (NC) describes a state in deep learning where intra-class features converge to their means, leading to maximum class separation. While significant, analyzing NC via traditional Shannon entropy is computationally prohibitive in high-dimensional spaces. This paper introduces matrix entropy as a robust alternative. By proposing two new metrics, the Matrix Mutual Information Ratio (MIR) and Matrix Entropy Difference Ratio (HDR), we provide tools to monitor training dynamics and develop a novel Cross-Model Alignment (CMA) loss. Our results demonstrate that these metrics effectively track NC and significantly improve performance in few-shot and semi-supervised learning tasks.

Introduction

In the terminal phase of training, deep neural networks often exhibit "Neural Collapse" (NC). In this state, intra-class features become nearly identical, and class means align with classifier weights, simplifying the model's geometric structure. Despite its importance, tracking NC is challenging. Traditional probability-based entropy requires estimating distributions in high-dimensional feature spaces, which is often computationally impractical. Furthermore, there is a lack of robust metrics to quantify the information interplay between data representations and classification heads, which is vital for understanding phenomena like "grokking."

Methodology: Matrix Information Theory

To overcome the limitations of Shannon entropy, we utilize Matrix Information Theory. Instead of probability distributions, we compute entropy based on the eigenvalues of a normalized Gram matrix, which represents feature similarities.

Conceptual overview of Matrix Entropy, MIR, and HDR calculation.

Core Metrics

  1. Matrix Entropy: Captures the information content of the feature space directly from the eigenvalues $\lambda_i$ of the Gram matrix.
  2. Matrix Mutual Information Ratio (MIR): Measures the shared information between features and weights.
  3. Matrix Entropy Difference Ratio (HDR): Measures the discrepancy in information levels between features and weights.

Theoretically, as a model approaches Neural Collapse, HDR should converge to 0, and MIR should approach 1.

Cross-Model Alignment (CMA) Loss

We propose the CMA loss to minimize intra-class matrix entropy across different modalities (e.g., image and text). Unlike global entropy reduction methods that risk "class collapse," CMA encourages tighter clustering within classes while maintaining inter-class separation.

Experimental Results

The proposed metrics and loss function were evaluated across standard benchmarks (CIFAR-10/100), few-shot learning (11 datasets), and semi-supervised learning (STL-10).

Neural Collapse Verification

Experimental observations confirm our theoretical derivations. As training progresses and the model reaches the NC state, MIR and HDR values stabilize toward their predicted limits.

Training dynamics of Accuracy, HDR, and MIR on CIFAR-10 and CIFAR-100.

Performance Gains

  • Few-Shot Learning: CMA consistently outperformed baseline and other entropy-based methods (like VNE) across diverse datasets using the CoOp framework.
  • Semi-Supervised Learning: Integrating our matrix-based loss into the FreeMatch algorithm significantly reduced error rates. On CIFAR-10 with 10 labels, the error rate dropped from 8% to 4.69%.
  • Grokking Signatures: MIR and HDR act as early indicators for grokking. A local minimum in MIR and a local maximum in HDR typically precede the sudden jump in generalization performance.

Conclusion and Future Work

Matrix entropy provides a computationally efficient and theoretically grounded framework for analyzing Neural Collapse. While eigenvalue decomposition poses some scaling challenges for extremely large batches, the proposed MIR and HDR metrics offer deep insights into model training dynamics. Future work will focus on extending the theoretical framework beyond MSE loss and developing adaptive weighting schemes for the CMA loss.

Summary updated

Abstract

In this paper, we introduce matrix entropy as an analytical tool for studying supervised learning, investigating the information content of data representations and classification head vectors, as well as the dynamic interactions between them during the supervised learning process. Our experimental results reveal that matrix entropy effectively captures the variations in information content of data representations and classification head vectors as neural networks approach Neural Collapse during supervised training, while also serving as a robust metric for measuring similarity among data samples. Leveraging this property, we propose Cross-Model Alignment (CMA) loss to optimize the fine-tuning of pretrained models. To characterize the dynamics of neural networks nearing the Neural Collapse state, we introduce two novel metrics: the Matrix Mutual Information Ratio (MIR) and the Matrix Entropy Difference Ratio (HDR), which quantitatively assess the interactions between data representations and classification heads in supervised learning, with theoretical optimal values derived under the Neural Collapse state. Our experiments demonstrate that MIR and HDR effectively explain various phenomena in neural networks, including the dynamics of standard supervised training, linear mode connectivity. Moreover, we use MIR and HDR to analyze the dynamics of grokking, which is a fascinating phenomenon in supervised learning where a model unexpectedly exhibits generalization long after achieving training data fit.
BibTeX
@misc{song2024exploring,
  author = {Kun Song and Zhiquan Tan and Bochao Zou and Jiansheng Chen and Huimin Ma and Weiran Huang},
  title = {{Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training}},
  year = {2024},
  eprint = {2409.16767},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.16767},
}
Figure