Abstract
The Laplace approximation provides a closed-form model selection objective for neural networks (NN). Online variants, which optimise NN parameters jointly with hyperparameters, like weight decay strength, have seen renewed interest in the Bayesian deep learning community. However, these methods violate Laplace's method's critical assumption that the approximation is performed around a mode of the loss, calling into question their soundness. This work re-derives online Laplace methods, showing them to target a variational bound on a mode-corrected variant of the Laplace evidence which does not make stationarity assumptions. Online Laplace and its mode-corrected counterpart share stationary points where 1. the NN parameters are a maximum a posteriori, satisfying the Laplace method's assumption, and 2. the hyperparameters maximise the Laplace evidence, motivating online methods. We demonstrate that these optima are roughly attained in practise by online algorithms using full-batch gradient descent on UCI regression datasets. The optimised hyperparameters prevent overfitting and outperform validation-based early stopping.
Introduction
In Bayesian deep learning, model selection often relies on estimating the marginal likelihood or "model evidence." The Laplace approximation is a popular tool for this, providing a tractable closed-form objective. However, its validity traditionally hinges on the neural network being at a local minimum (a "mode") of the loss function. Modern "Online Laplace" (OL) methods challenge this by updating hyperparameters concurrently with network weights. This creates a theoretical tension: can we trust hyperparameter updates when the stationarity assumption is violated? This paper reconciles this gap by framing OL as a variational lower bound.
Theoretical Framework: The Variational Link
The authors investigate the relationship between three objectives:
- Standard Online Laplace (OL): A heuristic that ignores first-order gradients during hyperparameter updates.
- Online Linear Model (LM): A "mode-corrected" version incorporating a first-order Taylor expansion (the gradient), representing the evidence of a tangent linear model.
- The Variational ELBO: The core contribution is proving that the OL objective is a Variational Lower Bound (ELBO) on the LM evidence.
By treating the current network parameters as a valid variational choice for the expansion point, the authors show that OL and LM share the same stationary points. Specifically, at convergence, the network reaches a Maximum A Posteriori (MAP) estimate, and the hyperparameters maximize the Laplace evidence.
Experimental Results
The researchers evaluated OL and LM on UCI regression datasets (e.g., Housing, Concrete, Energy) using single-hidden-layer MLPs.
Stability and Convergence
A key finding is that the "slack" (the gap) in the OL variational bound acts as a stabilizer. While the LM objective is tighter, it often leads to aggressive, unstable hyperparameter steps. In contrast, OL provides smoother convergence. As shown in the results, the distance between the neural network weights ($w_t$) and the tangent model's MAP ($v^*$) shrinks over time, indicating that the online process successfully navigates toward the theoretically sound offline optimum.
Predictive Performance
Online model selection consistently outperformed traditional offline methods using validation-based early stopping. The model evidence proved to be a more sensitive signal for hyperparameter tuning than simple validation loss, leading to superior test log-likelihoods across the benchmark datasets.
Conclusion
This work provides a theoretical foundation for Online Laplace methods, justifying their use despite the violation of classical stationarity assumptions. By demonstrating that OL targets a variational bound on a mode-corrected evidence, the authors explain its empirical success and stability. Future work will focus on scaling these methods to larger architectures and mini-batch training regimes.
Summary updated