TL;DR
Online Laplace (OL) methods tune neural network hyperparameters—such as weight decay and observation noise—on the fly during training, but they seem theoretically flawed because they discard the non-zero gradient term required away from a loss minimum. This paper proves that standard OL actually optimizes a principled variational lower bound (ELBO) on the evidence of a local tangent linear model. Crucially, the "slack" in this lower bound acts as a stabilizer, preventing the erratic updates that plague exact first-order corrected approaches.
What Problem Does This Paper Address?
Tuning hyperparameters (e.g., prior precision $\alpha$, noise precision $\beta$) via cross-validation is computationally expensive. Laplace approximations offer an alternative: they estimate the marginal likelihood (model evidence) in closed form by fitting a Gaussian around a local loss mode.
To avoid training models from scratch repeatedly, Online Laplace (OL) interleaves standard parameter updates with hyperparameter updates that maximize the Laplace evidence at the current weight vector $w_t$. However, this introduces a theoretical dilemma: Laplace’s method relies on a second-order Taylor expansion where the first-order gradient term is assumed to be zero (i.e., at a local mode). Mid-training, the gradient is non-zero. Simply dropping this first-order term seems mathematically unsound, yet OL works well in practice. This paper resolves this paradox.
Methods
The authors re-frame online hyperparameter tuning through the lens of a tangent linear model $h(v) = f(w_t) + J(v - w_t)$ linearized around current network weights $w_t$:
- Mode-Corrected Linear Model (LM): By keeping the first-order Taylor term rather than dropping it, the resulting objective $\mathcal{L}_h(M; w_t)$ corresponds to the exact log-marginal likelihood of the tangent linear model, effectively mode-corrected by an implicit Gauss-Newton step $v^*$.
- Online Laplace as an ELBO: The authors show that the standard OL objective $\mathcal{L}_f(M; w_t)$ is a variational evidence lower bound (ELBO) on $\mathcal{L}_h(M; w_t)$. Specifically, setting the variational posterior mean $\mu = v^*$ yields the exact tangent model evidence, while setting $\mu = w_t$ recovers standard Online Laplace.
- Shared Fixed Points: At convergence (when the network reaches a local optimum and $\partial_w \ell_f = 0$), the first-order term vanishes, the ELBO becomes tight ($\mathcal{L}_f = \mathcal{L}_h$), and the parameters match the tangent model's MAP.
Datasets
The authors evaluate their methods on standard UCI regression benchmarks using single-hidden-layer multilayer perceptrons (50 hidden units):
- Datasets:
housing,concrete,energy,wine, andyacht. - No new benchmarks are introduced.
Key Insights and Ideas
- Soundness Restored: Discarding the first-order gradient term is not an unjustified heuristic; it mathematically translates to optimizing a valid variational lower bound on the evidence of a local tangent linear model.
- Slack as a Feature, Not a Bug: While the mode-corrected approach (LM) uses a tighter bound, it produces overly aggressive hyperparameter steps that cause training instabilities. The slack in the standard OL bound dampens updates, acting as implicit regularization that stabilizes trajectory dynamics.
- Evidence vs. Generalization During Training: The model evidence peaks very early in online training, long before optimal test root mean squared error (RMSE) is reached. This confirms that local tangent evidence only aligns with neural network generalization once the parameters converge.
Key Results
- Predictive Performance: Online Laplace consistently achieved the highest or near-highest test log-likelihood across all five UCI datasets, outperforming offline networks trained with validation-based early stopping.
- Stability: Online LM performed comparably on
housing,concrete, andwine, but suffered severe instability onenergyandyacht(e.g., test log-likelihood dropping past $-6$ onyacht, whereas OL maintained around $0$). - Convergence to Shared MAP: In the online setting, the network weights $w_t$ and the tangent linear model MAP $v^*$ smoothly converged toward matching distributions, a property not observed during standard offline training.
Limitations and Future Work
- Scalability: Experiments were restricted to small MLPs (50 hidden units) and full-batch Adam. Calculating the Generalized Gauss-Newton (GGN) matrix and its log-determinant scales cubically with parameter or data dimensions, requiring structured approximations (e.g., K-FAC or subspace methods) for large-scale architectures.
- Non-Monotonic Dynamics: Because updating neural network weights alters the linear expansion point, the evidence landscape shifts continuously, meaning ELBO optimization is not guaranteed to be monotonic during intermediate training steps.
Summary updated