Unveiling the Dynamics of Information Interplay in Supervised Learning

Kun Song, Zhiquan Tan, Bochao Zou, Huimin Ma, Weiran Huang

Deep learning models are often treated as "black boxes." While their performance is undeniable, the internal mechanisms governing how features align with classification decisions remain poorly understood. This research introduces two novel information-theoretic metrics—Matrix Mutual Information Ratio (MIR) and Matrix Entropy Difference Ratio (HDR)—to quantify the interplay between data representations and model weights.

Quantifying Information Interplay

Traditional measures of model behavior often rely on simple similarity metrics or hard-to-estimate probability distributions. By leveraging Matrix Information Theory, the authors propose calculating entropy and mutual information directly from Gram matrices.

  • MIR (Matrix Mutual Information Ratio): Quantifies the information shared between data features and the classification head.
  • HDR (Matrix Entropy Difference Ratio): Measures the discrepancy in information content between these two components.

The Signature of Neural Collapse

A key contribution of this work is identifying the "information signature" of Neural Collapse—a phenomenon where features of the same class converge to a single point. As a model matures during training, MIR consistently trends toward its theoretical maximum, while HDR decreases toward zero. This provides a clear, quantifiable trajectory for healthy supervised training.

Figure 1: Evolution of Accuracy and MIR during training on CIFAR-10 and CIFAR-100. MIR tracks closely with accuracy, providing a quantifiable trajectory for model convergence.

Insights into Grokking and Pruning

The metrics offer new perspectives on complex training dynamics:

  • Grokking: The transition from memorization to generalization is marked by a distinct two-phase signature. Initially, MIR increases as the model fits the data, but then it decreases as the model shifts toward more efficient, generalized representations.
  • Pruning Resilience: Pruned models maintain high accuracy because their features retain a high MIR with the original unpruned model, preserving essential information despite significant parameter reduction.

Performance Gains in Semi-Supervised Learning

The practical utility of MIR and HDR is demonstrated by integrating them as auxiliary loss terms. In semi-supervised settings, where labeled data is scarce, these metrics act as powerful regularizers. By forcing the model to align the information of unlabeled samples with class weights, researchers observed dramatic improvements:

  • CIFAR-10 (10 labels): Error rates dropped from 8.07% to 4.69%.
  • STL-10 (40 labels): Error rates dropped from 15.56% to 12.10%.

Probing the Information Landscape

The authors also explored Linear Mode Connectivity, interpolating between two independently trained models. The behavior of MIR and HDR in these scenarios reveals nuances in the loss landscape, particularly when models are near their starting points or trained with different learning rates.

Figure 2: Accuracy, MIR, and HDR during model interpolation (lr: 3e-3). The metrics reflect the drop in performance between modes, highlighting the sensitivity of the information interplay to weight configuration.

Conclusion and Future Work

MIR and HDR provide a robust mathematical framework for understanding and improving deep learning. While computational complexity remains a challenge for massive datasets, these metrics open new avenues for explaining model behavior and enhancing performance in data-constrained environments. Future research will focus on extending these tools to Transformer architectures and Large Language Models (LLMs).

Summary updated

Abstract

In this paper, we use matrix information theory as an analytical tool to analyze the dynamics of the information interplay between data representations and classification head vectors in the supervised learning process. Specifically, inspired by the theory of Neural Collapse, we introduce matrix mutual information ratio (MIR) and matrix entropy difference ratio (HDR) to assess the interactions of data representation and class classification heads in supervised learning, and we determine the theoretical optimal values for MIR and HDR when Neural Collapse happens. Our experiments show that MIR and HDR can effectively explain many phenomena occurring in neural networks, for example, the standard supervised training dynamics, linear mode connectivity, and the performance of label smoothing and pruning. Additionally, we use MIR and HDR to gain insights into the dynamics of grokking, which is an intriguing phenomenon observed in supervised training, where the model demonstrates generalization capabilities long after it has learned to fit the training data. Furthermore, we introduce MIR and HDR as loss terms in supervised and semi-supervised learning to optimize the information interactions among samples and classification heads. The empirical results provide evidence of the method's effectiveness, demonstrating that the utilization of MIR and HDR not only aids in comprehending the dynamics throughout the training process but can also enhances the training procedure itself.
BibTeX
@inproceedings{song2024unveiling,
  author = {Kun Song and Zhiquan Tan and Bochao Zou and Huimin Ma and Weiran Huang},
  title = {{Unveiling the Dynamics of Information Interplay in Supervised Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=5hfvLBgnNE},
}
Figure