Characterizing the Edge of Stability in Variational Training Without Priors

Juraj Marusic, Jonathan Wenger, Beau Coker, John P. CunninghamarXiv 2026 · arXiv

TL;DR

Training variational deep learning models without explicit priors relies on "implicit regularization" from the optimizer to generalize. This paper reveals that using fewer Monte Carlo samples to estimate the loss—often seen as a necessary computational sacrifice—actually acts as a powerful regularizer by pushing the model toward flatter minima, significantly improving out-of-distribution robustness.

What Problem Does This Paper Address?

Bayesian methods are the "gold standard" for uncertainty quantification, but they are notoriously difficult to apply to deep learning. A major hurdle is prior elicitation: how do you choose a meaningful prior distribution over millions of neural network weights? Because bad priors lead to underfitting, researchers often downweight the regularization (the KL-divergence term) or remove it entirely, relying instead on the "implicit bias" of the optimization process.

While we know that standard gradient descent (GD) tends to operate at the Edge of Stability (EoS)—where the "sharpness" of the loss landscape (the largest eigenvalue of the Hessian) hovers around $2/\eta$ (where $\eta$ is the learning rate)—we haven't understood how this changes in a variational setting. Specifically, how do the number of parameter samples ($M$) and the noise from those samples affect the final model's sharpness and its ability to generalize to new, shifted data?

Methods

The authors focus on Unregularized Variational Training. Instead of the standard ELBO objective, they minimize the expected loss: $$\bar{\ell}(\theta) = \mathbb{E}_{w \sim q_\theta}[ \ell(w) ]$$ where $q_\theta$ is a Gaussian distribution over weights. In practice, this expectation is approximated using $M$ Monte Carlo samples.

The paper provides a rigorous theoretical characterization of this process:

  1. Linear Models: They derive Theorem 1, which defines the maximal stable learning rates for both the mean and covariance parameters.
  2. Smooth Losses: They extend this to arbitrary $\beta$-smooth loss landscapes (Theorem 2), showing that the stability threshold is modified by the Signal-to-Noise Ratio (SNR) of the gradient estimates and the number of samples $M$.
  3. Modified EoS: They propose a new EoS threshold for variational training: $$\lambda_{max} \approx \frac{2}{\eta} \left( \frac{1}{1 + \frac{1}{M\gamma}} \right)$$ where $\gamma$ is the SNR.

Datasets

The authors validated their theory using:

  • Synthetic Toy Models: To verify the sharpness bounds and function regularity.
  • CIFAR-10 & CIFAR-100: For standard in-distribution (ID) training and evaluation.
  • CIFAR-10-C & CIFAR-100-C: Corrupted versions of the original test sets (e.g., blur, noise, brightness changes) used to measure out-of-distribution (OOD) robustness.

Key Insights and Ideas

  • The "Less is More" Paradox: In most of machine learning, more samples ($M$) mean a better estimate. Here, the authors show that smaller $M$ is actually better for generalization. Using fewer samples increases gradient noise, which forces the optimizer to find flatter regions of the loss landscape that can tolerate that noise.
  • Modified Edge of Stability: The paper proves that variational training operates at a lower sharpness than standard training. While standard GD hits a ceiling at $2/\eta$, variational training with $M=1$ can cut that sharpness significantly (often by half if the SNR is near 1).
  • Sampling as Implicit Regularization: The noise introduced by sampling parameters from the variational distribution is distinct from the noise introduced by minibatches in SGD. This "parameter noise" persists even when the model fits the data perfectly, preventing the sharpness from exploding.
Figure 1: Variational training with fewer parameter samples leads to better generalization. The number of parameter samples used to estimate the expected loss in variational training (without prior regularization) affects the sharpness of the minima at convergence. A smaller number of samples M leads to convergence to flatter minima, which correspond to more regular functions (see (a); c.f. [8]), thus explaining the observed improved generalization. In (b), we show the sharpness of the loss during standard and variational training of a ResNet on CIFAR-10. Notably, in the variational case the sharpness remains below the standard threshold of 2/learning rate. When evaluated on a test set with distribution-shift, the variational model generalizes better out-of-distribution.

Key Results

  • Sharpness Reduction: Experiments on ResNet-34 architectures confirm that variational training consistently achieves lower peak sharpness than standard training across various learning rates.
  • Empirical Fit: The theoretical prediction for sharpness as a function of $M$ matches empirical results almost perfectly (see Figure 3b). As $M$ increases, the model reverts to the standard (sharper) EoS regime.
  • Robustness Gains: On CIFAR-10-C, the variational model ($M=1$) achieved significantly higher OOD accuracy and lower Negative Log-Likelihood (NLL) than standard training, all while maintaining identical in-distribution performance.
  • Efficiency: Because the best results come from $M=1$, these benefits are achieved at the lowest possible computational cost for variational training.
Figure 3: (a) Sharpness is lower for variational training across learning rates. (b) As the number of samples $M$ increases, sharpness rises toward the standard $2/\eta$ limit, while accuracy (both ID and OOD) actually drops.

Limitations and Future Work

  • Smoothness Assumption: The theoretical proofs rely on the loss being $\beta$-smooth. While this is standard in EoS literature, it doesn't strictly account for the non-smoothness of ReLU activations (though the empirical results on ResNets suggest the theory holds up well).
  • Optimization Specifics: The current theory does not explicitly model the interaction with momentum or the specific noise characteristics of SGD minibatches.
  • Variational Families: The work is restricted to Gaussian variational families with low-rank covariance. Exploring more expressive families, like normalizing flows, is a suggested next step.
  • Practical Schedulers: The authors suggest that their stability bounds could be used to design more efficient learning rate schedulers specifically for variational training.

Summary updated

Abstract

Variational approaches in deep learning promise to deliver improved uncertainty quantification and out-of-distribution generalization by inferring a distribution over weights, regularized by a divergence to a chosen prior. However, prior elicitation presents a significant challenge in deep learning, resulting in the common practice of significantly downweighting the regularization term in the variational objective. Recently, it was proposed to train solely via the expected loss by relying on implicit regularization from the optimizer, rather than explicit regularization from the divergence to the prior. While this approach largely sidesteps the issue of prior elicitation, current theory lacks an understanding of how key optimization hyperparameters impact the implicit regularization. Here, we theoretically and empirically demonstrate that larger learning rates and fewer parameter samples lead to flatter minima of the loss landscape, which are known to generalize better. We show that training on the expected loss operates at a modified edge of stability, characterized by the signal-to-noise ratio of the gradient estimates and the number of Monte Carlo samples. Experiments on out-of-distribution benchmark datasets confirm the practical significance of the theoretical results.
BibTeX
@misc{marusic2026characterizing,
  author = {Juraj Marusic and Jonathan Wenger and Beau Coker and John P. Cunningham},
  title = {{Characterizing the Edge of Stability in Variational Training Without Priors}},
  year = {2026},
  eprint = {2611.15990},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15990},
}
Figure