TL;DR
Training variational deep learning models without explicit priors relies on "implicit regularization" from the optimizer to generalize. This paper reveals that using fewer Monte Carlo samples to estimate the loss—often seen as a necessary computational sacrifice—actually acts as a powerful regularizer by pushing the model toward flatter minima, significantly improving out-of-distribution robustness.
What Problem Does This Paper Address?
Bayesian methods are the "gold standard" for uncertainty quantification, but they are notoriously difficult to apply to deep learning. A major hurdle is prior elicitation: how do you choose a meaningful prior distribution over millions of neural network weights? Because bad priors lead to underfitting, researchers often downweight the regularization (the KL-divergence term) or remove it entirely, relying instead on the "implicit bias" of the optimization process.
While we know that standard gradient descent (GD) tends to operate at the Edge of Stability (EoS)—where the "sharpness" of the loss landscape (the largest eigenvalue of the Hessian) hovers around $2/\eta$ (where $\eta$ is the learning rate)—we haven't understood how this changes in a variational setting. Specifically, how do the number of parameter samples ($M$) and the noise from those samples affect the final model's sharpness and its ability to generalize to new, shifted data?
Methods
The authors focus on Unregularized Variational Training. Instead of the standard ELBO objective, they minimize the expected loss: $$\bar{\ell}(\theta) = \mathbb{E}_{w \sim q_\theta}[ \ell(w) ]$$ where $q_\theta$ is a Gaussian distribution over weights. In practice, this expectation is approximated using $M$ Monte Carlo samples.
The paper provides a rigorous theoretical characterization of this process:
- Linear Models: They derive Theorem 1, which defines the maximal stable learning rates for both the mean and covariance parameters.
- Smooth Losses: They extend this to arbitrary $\beta$-smooth loss landscapes (Theorem 2), showing that the stability threshold is modified by the Signal-to-Noise Ratio (SNR) of the gradient estimates and the number of samples $M$.
- Modified EoS: They propose a new EoS threshold for variational training: $$\lambda_{max} \approx \frac{2}{\eta} \left( \frac{1}{1 + \frac{1}{M\gamma}} \right)$$ where $\gamma$ is the SNR.
Datasets
The authors validated their theory using:
- Synthetic Toy Models: To verify the sharpness bounds and function regularity.
- CIFAR-10 & CIFAR-100: For standard in-distribution (ID) training and evaluation.
- CIFAR-10-C & CIFAR-100-C: Corrupted versions of the original test sets (e.g., blur, noise, brightness changes) used to measure out-of-distribution (OOD) robustness.
Key Insights and Ideas
- The "Less is More" Paradox: In most of machine learning, more samples ($M$) mean a better estimate. Here, the authors show that smaller $M$ is actually better for generalization. Using fewer samples increases gradient noise, which forces the optimizer to find flatter regions of the loss landscape that can tolerate that noise.
- Modified Edge of Stability: The paper proves that variational training operates at a lower sharpness than standard training. While standard GD hits a ceiling at $2/\eta$, variational training with $M=1$ can cut that sharpness significantly (often by half if the SNR is near 1).
- Sampling as Implicit Regularization: The noise introduced by sampling parameters from the variational distribution is distinct from the noise introduced by minibatches in SGD. This "parameter noise" persists even when the model fits the data perfectly, preventing the sharpness from exploding.
Key Results
- Sharpness Reduction: Experiments on ResNet-34 architectures confirm that variational training consistently achieves lower peak sharpness than standard training across various learning rates.
- Empirical Fit: The theoretical prediction for sharpness as a function of $M$ matches empirical results almost perfectly (see Figure 3b). As $M$ increases, the model reverts to the standard (sharper) EoS regime.
- Robustness Gains: On CIFAR-10-C, the variational model ($M=1$) achieved significantly higher OOD accuracy and lower Negative Log-Likelihood (NLL) than standard training, all while maintaining identical in-distribution performance.
- Efficiency: Because the best results come from $M=1$, these benefits are achieved at the lowest possible computational cost for variational training.
Limitations and Future Work
- Smoothness Assumption: The theoretical proofs rely on the loss being $\beta$-smooth. While this is standard in EoS literature, it doesn't strictly account for the non-smoothness of ReLU activations (though the empirical results on ResNets suggest the theory holds up well).
- Optimization Specifics: The current theory does not explicitly model the interaction with momentum or the specific noise characteristics of SGD minibatches.
- Variational Families: The work is restricted to Gaussian variational families with low-rank covariance. Exploring more expressive families, like normalizing flows, is a suggested next step.
- Practical Schedulers: The authors suggest that their stability bounds could be used to design more efficient learning rate schedulers specifically for variational training.
Summary updated