Data-Free Metrics Don't Work: The Missing Invariance Under Functionally Preserving Reparameterisations

Gabryel Mason-Williams, Israel Mason-Williams, Fredrik DahlqvistarXiv 2026 · arXiv

TL;DR

This paper debunked the popular claim that data-free metrics—specifically those based on Random Matrix Theory (RMT)—can reliably predict a neural network's generalization performance. By showing that these metrics can be changed arbitrarily without affecting a model's actual function, the authors demonstrate that the supposed link between these "magic numbers" and model quality is largely a spurious artifact of initialization and training procedures.

What Problem Does This Paper Address?

In the quest for interpretable AI, researchers have sought "data-free" metrics: ways to judge if a model is well-trained or "healthy" just by looking at its weights, without needing a test dataset. Prominent among these is the theory of Heavy-Tailed Self-Regularization (HTSR), which uses RMT to analyze the spectra of weight matrices. It suggests that a specific power-law exponent ($\alpha$) falling in a "good" range (typically between 2 and 6) indicates a well-generalizing model.

These metrics have been used to guide everything from pruning and compression to choosing learning rates. However, this paper asks a fundamental scientific question: Is there a causal link between these metrics and performance, or are we just seeing a correlation? If a metric truly captures "well-trainedness," it should be invariant—it shouldn't change if we modify the model in a way that preserves its functional behavior.

Methods

The authors employ a clever strategy inspired by the "robust vs. critical" layer phenomenon. In any trained network, some layers are critical (changing them destroys performance), while others are robust (they can be reset to their initial values or even randomized without significantly hurting the model's accuracy).

To test the RMT metrics, the authors:

  1. Constructed Functionally Preserving Reparameterisations: They took pre-trained models and re-initialized or re-randomized their robust layers. This creates a new set of weights that performs nearly identically to the original.
  2. Measured Invariance: They checked if the $\alpha$ metric remained stable under these changes.
  3. Controlled Training: They trained models while forcing the weight spectra to stay within "bad" $\alpha$ ranges to see if it actually prevented generalization.
  4. SVD Ablation: They used Singular Value Decomposition (SVD) to isolate the effects of singular values (which RMT metrics measure) versus singular vectors (the "directions" of the weights) during training.

Datasets

The study spans multiple scales and modalities:

  • Vision: MNIST (using a 5-layer ReLU FCN) and ImageNet (using ResNet34 and Vision Transformers).
  • Language: WikiText-103 (using GPT-2, GPT-2 Large, and Pythia-1B).

Key Insights and Ideas

  • The Invariance Failure: The most striking "aha moment" is that RMT metrics are not reparameterisation-invariant. You can take a robust layer, re-randomize it, and watch its $\alpha$ value fly off into "underfitted" territory while the model's total accuracy barely budges.
  • Good Metrics $\neq$ Good Models: As shown in Figure 2, many networks whose performance is destroyed by re-randomization still maintain "good" $\alpha$ values. Conversely, models can maintain high accuracy even when their layers have "bad" metrics.
  • Initialization is the Real Driver: The authors argue that if well-trained models often have $\alpha \in [2, 6]$, it's not because that range is a prerequisite for generalization. Instead, it's a side effect of standard initialization schemes and the regularizing nature of Stochastic Gradient Descent (SGD).
  • The "Signs" Matter More: In their analysis of the Lottery Ticket Hypothesis (Appendix H), they found that the signs of the weights (the pathways they create) are far more important for preserving function than the spectral properties measured by RMT.
Figure 2: (a) Empirical distribution of α values on a 512x512 fully connected layer, sampled from 10,000 initialisations. (b) Layer re-initialisation (•) and re-randomisation (•) test accuracy vs α. In § D we present the descriptive statistics for sub-figure (a) along with other initialisation schemes.

Key Results

  • Zero Correlation: Across ResNet, ViT, and GPT-2 architectures, the Spearman correlation between the $\alpha$ metric and test accuracy was effectively zero (e.g., 0.006 for ResNet34).
  • Arbitrary Training: The authors successfully trained MNIST models with fixed $\alpha$ values in "bad" ranges (e.g., $18 \le \alpha \le 20$) and still achieved ~90% accuracy, identical to models trained in the "optimal" range.
  • Failure to Disambiguate: Data-free metrics could not distinguish between robust and critical layers, nor could they predict the performance drop when a layer was tampered with.
Figure 3 & 4: Scatter plots for ResNet34 and GPT-2 show no consistent relationship between alpha and test accuracy after layer re-randomization.

Limitations and Future Work

The authors acknowledge that while they tested the most prominent RMT-based metrics, they haven't exhausted every possible data-free metric in existence. They also focused on vision and language, leaving other domains like audio or reinforcement learning for future validation.

The core takeaway is a call to action: the field should move away from metrics that only look at weight distributions in isolation. Instead, we need "data-based" metrics that account for the nuanced interplay between weights and the data they process. They suggest that future interpretability research should prioritize metrics that are actually invariant under functional preservation, providing a more rigorous ground for understanding how deep networks truly generalize.

Summary updated

Abstract

Data-free methods for analysing and understanding the layers of neural networks offer many metrics for quantifying notions of 'strong' versus 'weak' layers, with the promise of increased interpretability. In particular, random matrix theory (RMT) offers data-free metrics that claim predictive power over the quality of pre-trained models at a layerwise precision and the ability to identify model pathologies, indicating a unique relationship with generalisation. As a result, metrics from RMT are championed for pre-training optimisation strategies and post-training compression. We establish that RMT-based metrics are unrelated to performance or training by showing that functionally preserving reparametrisations of a pre-trained model can have arbitrary metrics. We show this across a range of architectures and scales. To create functionally preserving reparametrisations, we exploit the well-established phenomenon of criticality: some layers can be re-initialised or re-randomised without affecting the functional behaviour of the model -- they are called robust -- while others cannot -- they are called critical. Re-initialising or re-randomising robust layers provides functionally preserving reparametrisations for which RMT-based metrics are arbitrary. Moreover, we show that relationships between metrics offered by RMT are spuriously related to generalisation. We conclude by showing that if many pre-trained models have data-free metrics in a 'good' range, it is, in part, dependent on model initialisation. Project page available at: https://gmw99.github.io/data-free-metrics-dont-work
BibTeX
@misc{masonwilliams2026data,
  author = {Gabryel Mason-Williams and Israel Mason-Williams and Fredrik Dahlqvist},
  title = {{Data-Free Metrics Don't Work: The Missing Invariance Under Functionally Preserving Reparameterisations}},
  year = {2026},
  eprint = {2611.15621},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15621},
}
Figure