Deep learning models are often treated as "black boxes." While their performance is undeniable, the internal mechanisms governing how features align with classification decisions remain poorly understood. This research introduces two novel information-theoretic metrics—Matrix Mutual Information Ratio (MIR) and Matrix Entropy Difference Ratio (HDR)—to quantify the interplay between data representations and model weights.
Quantifying Information Interplay
Traditional measures of model behavior often rely on simple similarity metrics or hard-to-estimate probability distributions. By leveraging Matrix Information Theory, the authors propose calculating entropy and mutual information directly from Gram matrices.
- MIR (Matrix Mutual Information Ratio): Quantifies the information shared between data features and the classification head.
- HDR (Matrix Entropy Difference Ratio): Measures the discrepancy in information content between these two components.
The Signature of Neural Collapse
A key contribution of this work is identifying the "information signature" of Neural Collapse—a phenomenon where features of the same class converge to a single point. As a model matures during training, MIR consistently trends toward its theoretical maximum, while HDR decreases toward zero. This provides a clear, quantifiable trajectory for healthy supervised training.
Insights into Grokking and Pruning
The metrics offer new perspectives on complex training dynamics:
- Grokking: The transition from memorization to generalization is marked by a distinct two-phase signature. Initially, MIR increases as the model fits the data, but then it decreases as the model shifts toward more efficient, generalized representations.
- Pruning Resilience: Pruned models maintain high accuracy because their features retain a high MIR with the original unpruned model, preserving essential information despite significant parameter reduction.
Performance Gains in Semi-Supervised Learning
The practical utility of MIR and HDR is demonstrated by integrating them as auxiliary loss terms. In semi-supervised settings, where labeled data is scarce, these metrics act as powerful regularizers. By forcing the model to align the information of unlabeled samples with class weights, researchers observed dramatic improvements:
- CIFAR-10 (10 labels): Error rates dropped from 8.07% to 4.69%.
- STL-10 (40 labels): Error rates dropped from 15.56% to 12.10%.
Probing the Information Landscape
The authors also explored Linear Mode Connectivity, interpolating between two independently trained models. The behavior of MIR and HDR in these scenarios reveals nuances in the loss landscape, particularly when models are near their starting points or trained with different learning rates.
Conclusion and Future Work
MIR and HDR provide a robust mathematical framework for understanding and improving deep learning. While computational complexity remains a challenge for massive datasets, these metrics open new avenues for explaining model behavior and enhancing performance in data-constrained environments. Future research will focus on extending these tools to Transformer architectures and Large Language Models (LLMs).
Summary updated