More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize

Alexander Wei, Wei Hu, Jacob Steinhardt

Abstract

Deep learning presents a fundamental challenge to classical statistics: models with millions of parameters generalize remarkably well even when they have the capacity to perfectly memorize training data. While traditional generalization bounds based on weight norms or covariance spectra often become "vacuous" in high-dimensional settings, this paper demonstrates that Random Matrix Theory (RMT) provides a powerful framework for predicting real-world performance. By leveraging the Generalized Cross-Validation (GCV) estimator, the authors accurately predict the generalization risk of large-scale ResNet models and resolve the "pretraining paradox," showing that pretrained models succeed not through simplicity, but through superior feature alignment.

Introduction

In modern machine learning, the gap between theoretical generalization bounds and empirical performance is often vast. In high-dimensional regimes—such as those involving ResNet architectures trained on ImageNet—classical norm-based bounds frequently fail to provide useful information, sometimes even predicting that error will increase as more data is added. This discrepancy suggests that our current theoretical tools are missing a fundamental aspect of how neural networks learn.

This research argues that Random Matrix Theory (RMT), often dismissed as a simplified "toy" framework, is actually capable of mapping onto the complexities of real-world neural representations. By focusing on Kernel Ridge Regression (KRR) applied to the empirical Neural Tangent Kernel (eNTK) of deep networks, the authors provide a method to predict test error using only training data, offering a more precise lens into the mechanics of generalization.

Methodology: GCV and the RMT Framework

The technical core of this study is the Generalized Cross-Validation (GCV) estimator. Although GCV has been a staple of statistics since the 1970s, the authors provide a modern theoretical justification for its efficacy in deep learning using RMT. They prove that GCV converges to the true generalization risk whenever a "local random matrix law" (a variant of the Marchenko-Pastur law) holds for the feature covariance.

To understand why models generalize, the authors decompose the generalization risk into two primary factors:

  1. Eigendecay ((\gamma)): A measure of how quickly the importance of different feature dimensions drops off.
  2. Alignment ((\delta)): A measure of how well the "ground truth" function matches the top eigenvectors of the feature space.

By analyzing these factors across various ResNet architectures (from ResNet-18 to ResNet-101) and datasets including CIFAR-10, CIFAR-100, and Food-101, the researchers were able to test the predictive power of RMT in diverse settings.

The Pretraining Paradox: Complexity vs. Alignment

One of the paper's most significant contributions is the resolution of the "pretraining paradox." Conventional wisdom suggests that pretraining works because it encourages "simpler," lower-dimensional representations that are less prone to overfitting. However, the authors' empirical analysis reveals the exact opposite: pretrained kernels actually exhibit slower eigendecay and a higher effective dimension than randomly initialized ones.

If pretrained models are more complex, why do they generalize better? The answer lies in alignment. Pretraining reshapes the feature space so that the target task is much easier to represent using the most prominent features. This benefit of alignment significantly outweighs the theoretical "cost" of a higher-dimensional representation. This finding suggests that the search for "simplicity" in neural networks may be less important than the search for task-relevant feature alignment.

Key Results and Empirical Validation

The RMT-based GCV estimator demonstrated remarkable predictive accuracy across various benchmarks:

  • High Predictive Accuracy: Over 90% of the test-risk predictions made by the GCV estimator had less than 0.09 error (in both relative and absolute terms) across different dataset sizes and regularization levels.
  • Failure of Traditional Baselines: In contrast, traditional norm-based predictors were often negatively correlated with actual test risk, incorrectly predicting performance degradation when the models were actually improving.
  • Scaling Law Predictions: By estimating the power-law exponents for eigendecay and alignment, the authors successfully predicted "scaling laws"—the rate at which performance improves as training data increases—for ridge regression on natural data.

Discussion and Future Directions

The study also challenges the distinction between noise and signal in high dimensions. The authors argue that in these spaces, label noise and complex ground-truth signals become mathematically indistinguishable, suggesting that "noiseless" regression models are sufficient for capturing the behavior of even noisy real-world datasets.

While this work masterfully handles linear models and kernels, several avenues for future research remain. Extending these RMT insights to non-linear fine-tuning, such as full gradient descent or logistic regression, is a primary challenge. Additionally, the current theory requires the regularization parameter ((\lambda)) to be relatively large; extending the framework to handle the "near-zero" regularization regime common in practice is a necessary next step. Ultimately, the use of eNTKs as a tractable model provides a promising path toward a more rigorous theory of feature learning in deep neural networks.

Summary updated

BibTeX
@inproceedings{wei2022more,
  author = {Alexander Wei and Wei Hu and Jacob Steinhardt},
  title = {{More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=cCdrvWnm4X},
}
Figure