Abstract
In many applications, one works with neural network models trained by someone else. For such pretrained models, one may not have access to training data or test data. Moreover, one may not know details about the model, e.g., the specifics of the training data, the loss function, the hyperparameter values, etc. Given one or many pretrained models, it is a challenge to say anything about the expected performance or quality of the models. Here, we address this challenge by providing a detailed meta-analysis of hundreds of publicly-available pretrained models. We examine norm based capacity control metrics as well as power law based metrics from the recently-developed Theory of Heavy-Tailed Self Regularization. We find that norm based metrics correlate well with reported test accuracies for well-trained models, but that they often cannot distinguish well-trained versus poorly-trained models. We also find that power law based metrics can do much better -- quantitatively better at discriminating among series of well-trained models with a given architecture; and qualitatively better at discriminating well-trained versus poorly-trained models. These methods can be used to identify when a pretrained neural network has problems that cannot be detected simply by examining training/test accuracies.
Introduction
Evaluating the performance of pretrained neural networks is a significant challenge when access to training data, test sets, or training details (e.g., loss functions, hyperparameters) is unavailable. Standard evaluation relies on test accuracy, which may not be accessible or may fail to detect structural issues like over-optimization or brittleness. This work presents a meta-analysis of hundreds of publicly available models, demonstrating that internal weight matrix structures can predict generalization performance and identify training pathologies.
Methodology: Heavy-Tailed Self-Regularization
The proposed approach utilizes the Theory of Heavy-Tailed Self-Regularization (HT-SR) to analyze the Empirical Spectral Density (ESD) of model weight matrices. The process, implemented in the WeightWatcher tool, involves several key steps:
- Spectral Analysis: For each layer, the weight matrix $W$ is used to compute the correlation matrix $X = W^T W$.
- Eigenvalue Distribution: The eigenvalues ($\lambda$) of $X$ are calculated to form an ESD, $\rho(\lambda)$.
- Power Law Fitting: The ESD is fitted to a power-law distribution, $\rho(\lambda) \sim \lambda^{-\alpha}$, where $\alpha$ is the power-law exponent.
The study compares traditional norm-based capacity metrics (e.g., Frobenius and Spectral norms) against Power Law (PL) metrics, such as the Weighted Alpha ($\hat{\alpha}$), which aggregates $\alpha$ values across layers. Well-trained models typically exhibit heavy-tailed ESDs with $\alpha$ values between 2 and 4.
Experimental Results
The meta-analysis covered a wide range of architectures, including VGG, ResNet, and GPT series, trained on benchmarks like ImageNet and CIFAR.
Predicting Performance Trends
For well-trained model series like VGG, both norm-based and PL-based metrics correlate strongly with reported test accuracies. However, PL-based metrics demonstrate superior robustness across more complex architectures. For the ResNet-1K series, Weighted Alpha achieved an $R^2$ of 0.89, significantly outperforming the Spectral Norm ($R^2 = 0.74$).
Diagnostic Capabilities: Scale Collapse
A key finding is the identification of "Scale Collapse," a phenomenon where model compression or distillation degrades the underlying correlation structure. While norm-based metrics might suggest these models are improved due to smaller norms, PL metrics reveal the structural damage. This allows for the detection of "brittle" models that may maintain high test accuracy but lack robust generalization.
Discussion and Conclusion
The results indicate that the heavy-tailed nature of weight matrices is a reliable indicator of model quality. Lower values of the PL exponent $\alpha$ generally signify better-trained models that have effectively captured data correlations. While the theory is currently semi-empirical, it provides a powerful "data-free" diagnostic tool for evaluating and guiding the training of state-of-the-art neural networks. Future work aims to refine these diagnostics to enable direct comparisons across diverse architecture families.
Summary updated