TPV: Parameter Perturbations Through the Lens of Test Prediction Variance

Devansh Arpit

Introduction

Deep learning theory has long sought a unified metric to explain why certain models generalize better than others. While concepts like "wide minima," the "implicit bias" of Stochastic Gradient Descent (SGD), and the "Neural Tangent Kernel" (NTK) offer valuable perspectives, they often remain disconnected. Furthermore, traditional bias-variance analysis typically focuses on global variance—the change in a model when retrained from scratch. In practice, however, we are often more concerned with local robustness: how a specific, trained model responds to small perturbations like quantization, pruning, or noise during fine-tuning.

This paper introduces Test Prediction Variance (TPV), a single, label-free metric that measures the sensitivity of a model's outputs to infinitesimal parameter perturbations. The authors demonstrate that TPV not only unifies various noise phenomena but can also be accurately estimated using only training data, even when a significant generalization gap exists.

The TPV Framework

The core of the TPV approach is a "trace form" approximation that separates the model's geometry from the specific noise mechanism:

$$TPV \approx \text{Tr}(H_{ ext{eff}} C)$$

In this equation, $H_{ ext{eff}}$ (Effective Curvature) represents the second moment of the Jacobian, capturing the local geometry of the trained model. $C$ (Perturbation Covariance) represents the noise mechanism, such as SGD noise or pruning masks. By decoupling these two components, TPV provides a versatile tool for analyzing how different types of perturbations affect model stability.

The Stability Surprise

Perhaps the most significant theoretical and empirical contribution of this work is the discovery of "TPV Trace Stability." Using NTK theory, the authors prove that in overparameterized networks, the TPV estimated on the training set converges to its value on the test set. This means that the geometric information contained within the training inputs is sufficient to predict how the model will react to noise on unseen data.

TPV stability: TPV estimated on the training set matches the test set across five orders of magnitude, regardless of model width.

As shown in the figure above, empirical tests across five orders of magnitude demonstrate that training and test TPV concentrate tightly around the $y=x$ diagonal. Remarkably, this stability holds even for models that are heavily overfitted, making TPV a reliable diagnostic tool that does not require a labeled validation set.

TPV as a Proxy for Generalization

Beyond measuring sensitivity, TPV serves as a powerful indicator of model performance. The research finds that "wide minima" are essentially regions characterized by low TPV. Whether the noise originates from finite-precision quantization or SGD, sharper minima lead to higher TPV and, consequently, poorer test performance.

Generalization correlation: Lower TPV consistently correlates with higher validation accuracy across diverse standard architectures.

Experiments on large-scale benchmarks like ImageNet confirm this relationship. As illustrated in the plot of validation accuracy versus empirical TPV, models with lower sensitivity (lower TPV) consistently achieve higher accuracy. This correlation persists across a wide variety of modern architectures, including ResNet, EfficientNet, and ConvNeXt.

Key Insights and Applications

  • Width as a Stabilizer: Increasing the width of a network improves the conditioning of the model's Jacobian. This suppresses TPV, providing a geometric explanation for why larger models tend to be more robust to label noise and generalize better.
  • JBR Pruning: The authors leverage the TPV framework to develop Jacobian-Based Rebalancing (JBR), a new pruning algorithm. By treating pruning as a parameter perturbation, JBR identifies and removes weights in a way that minimizes the resulting TPV. This method matches or exceeds state-of-the-art pruning baselines, particularly in high-sparsity regimes (e.g., 90% pruning on ImageNet).

Limitations and Future Directions

While TPV offers a robust framework, it currently relies on a first-order Taylor approximation (linearization). If perturbations are excessively large—such as in cases of extreme label noise—the model may move beyond the local regime where TPV stability is guaranteed. Additionally, while empirical results for Cross-Entropy loss are strong, the formal theory is currently most developed for Mean Squared Error (MSE). Future work aims to extend this framework to input perturbations (adversarial robustness) and broader distribution shifts.

Summary updated

BibTeX
@inproceedings{arpit2026tpv,
  author = {Devansh Arpit},
  title = {{TPV: Parameter Perturbations Through the Lens of Test Prediction Variance}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=HskjhAfdY4},
}
Figure