Abstract: We perform scalable approximate inference in continuous-depth Bayesian neural networks. In this model class, uncertainty about separate weights in each layer gives hidden units that follow a stochastic differential equation. We demonstrate gradient-based stochastic variational inference in this infinite-parameter setting, producing arbitrarily-flexible approximate posteriors. We also derive a novel gradient estimator that approaches zero variance as the approximate posterior over weights approaches the true posterior. This approach brings continuous-depth Bayesian neural nets to a competitive comparison against discrete-depth alternatives, while inheriting the memory-efficient training and tunable precision of Neural ODEs.
Introduction
Deep learning models typically follow one of two paradigms for advanced architectural design. Continuous-depth models, such as Neural ODEs, decouple model specification from computation, enabling memory-efficient training via the adjoint method and adaptive computation. Conversely, Bayesian Neural Networks (BNNs) maintain a distribution over weights rather than point estimates, providing inherent uncertainty quantification, improved calibration, and robustness to out-of-distribution data.
Naively combining these—for instance, by placing a standard Gaussian approximate posterior over the finite parameters of a Neural ODE—fails to fully exploit the continuous-time paradigm and often yields poor performance. This paper introduces SDE-BNNs, which address this by modeling an infinitely deep BNN where separate, uncertain weights exist at every infinitesimal layer. This results in hidden units that evolve according to a Stochastic Differential Equation (SDE).
Methodology
The SDE-BNN models the evolution of both the network's hidden states and its weights as a joint SDE.
Prior and Posterior as SDEs
Instead of standard Gaussian priors, the prior over the continuous weight trajectories is defined as an Ornstein–Uhlenbeck (OU) process. The approximate posterior is defined as another SDE with the same diffusion but a learned drift function, parameterized by a hypernetwork. By the Girsanov theorem, as long as the prior and posterior SDEs share the same diffusion function, the approximate posterior can be made arbitrarily expressive simply by increasing the capacity of the drift function's neural network.
Augmented State and Inference
To evaluate the network, the hidden units and weights are concatenated into a single augmented state. A black-box adaptive SDE solver integrates this system forward in time.
One major challenge in this setting is the high variance of standard Evidence Lower Bound (ELBO) gradient estimators for SDEs. The authors adapt the "sticking the landing" (STL) trick to continuous time. By dropping a zero-expectation score function term from the path-space KL divergence, they derive a novel gradient estimator whose variance vanishes as the approximate posterior approaches the true posterior. This stabilization is critical for optimizing infinite-dimensional variational objectives.
Key Results
The model was evaluated on synthetic regression and standard image classification benchmarks (MNIST, CIFAR-10, and CIFAR-10C).
1D Synthetic Regression
On a non-monotonic toy dataset, the SDE-BNN demonstrates its ability to learn highly flexible posteriors. Unlike some neural SDEs that inject noise directly into hidden states, SDE-BNNs inject noise into the weights. This ensures that a single sampled weight trajectory produces a deterministic, smooth mapping from input to output, which is essential for stable function approximation.
Accuracy, Calibration, and Robustness
- CIFAR-10: The SDE-BNN achieved ~89.8% accuracy, significantly outperforming a Mean-Field Neural ODE (81.5%). It also showed superior calibration, with an Expected Calibration Error (ECE) of $7.19 \times 10^{-2}$ compared to $8.71 \times 10^{-2}$ for standard Neural ODEs.
- Robustness (CIFAR-10C): When tested against 19 real-world image corruptions, the SDE-BNN achieved roughly 4.4% lower absolute corruption error compared to standard ResNet32 baselines, proving that its learned uncertainty translates to real-world robustness.
- Gradient Variance: The proposed STL gradient estimator reduced gradient variance by a factor of approximately 4 on toy problems compared to standard Monte Carlo estimators.
Limitations and Future Work
Despite its performance, the SDE-BNN is computationally intensive, as evaluation scales with the number of weights and solver iterations. Additionally, the model is currently incompatible with Batch Normalization, which breaks the Lipschitz continuity required for unique SDE solutions.
Future work involves regularizing these models to require fewer solver steps and applying the low-variance STL gradient estimator to other domains, such as variational inference in time-series modeling.
Summary updated