Abstract
We perform scalable approximate inference in continuous-depth Bayesian neural networks. In this model class, uncertainty about separate weights in each layer gives hidden units that follow a stochastic differential equation. We demonstrate gradient-based stochastic variational inference in this infinite-parameter setting, producing arbitrarily-flexible approximate posteriors. We also derive a novel gradient estimator that approaches zero variance as the approximate posterior over weights approaches the true posterior. This approach brings continuous-depth Bayesian neural nets to a competitive comparison against discrete-depth alternatives, while inheriting the memory-efficient training and tunable precision of Neural ODEs.
Introduction
Deep learning models often struggle with the trade-off between model depth and robust uncertainty quantification. While Neural ODEs offer memory-efficient, continuous-depth representations, they are typically deterministic point-estimate models. Conversely, Bayesian Neural Networks (BNNs) provide principled uncertainty estimates, but scaling them to "infinitely deep" architectures presents significant mathematical and computational challenges.
This paper introduces SDE-BNNs, a new class of Bayesian neural networks where weights evolve continuously with depth according to a Stochastic Differential Equation (SDE). This framework allows for an infinite-parameter model that remains trainable and scalable, combining the benefits of continuous-depth models with a flexible Bayesian treatment.
Methods: Continuous-Depth Bayesian Inference
The core of the SDE-BNN is treating hidden state transformations as a stochastic process.
Weight Dynamics and Priors
Instead of static weights, the weights $w_t$ follow an SDE. The authors employ an Ornstein-Uhlenbeck (OU) process as a prior. This choice is critical as it ensures that weight magnitudes remain bounded as depth increases, preventing numerical instability.
Variational Inference and the STL Estimator
To perform inference, the authors define an approximate posterior using another SDE, where the "drift" (the direction of weight movement) is learned via a separate neural network. A key technical contribution is the Sticking the Landing (STL) gradient estimator. By adapting this technique to the SDE setting, the authors derive a variance-reduced gradient that removes noisy terms with zero expectation, significantly stabilizing training.
Efficiency
Training is made efficient through the stochastic adjoint sensitivity method. This allows the model to be trained with constant memory cost relative to the number of solver steps, inheriting the primary advantage of Neural ODEs.
Experimental Results
The authors evaluated SDE-BNNs across synthetic regression and image classification benchmarks including MNIST, CIFAR-10, and CIFAR-10C.
Calibration and Robustness
The most significant advantage of SDE-BNNs is their calibration. On CIFAR-10, the SDE-BNN achieved an Expected Calibration Error (ECE) of 0.78%, outperforming standard ODE-Nets (~1.11%). Furthermore, in the CIFAR-10C benchmark—which tests robustness against corrupted data—SDE-BNNs outperformed ResNet32 and Mean Field BNNs by approximately 4.4% in absolute corruption error. This demonstrates that explicitly accounting for weight uncertainty leads to more reliable predictions in noisy, real-world conditions.
Variance Reduction
The STL gradient estimator was shown to reduce gradient variance by a factor of approximately 4 on toy problems, facilitating faster and more reliable convergence compared to standard Monte Carlo approaches.
Conclusion and Future Work
SDE-BNNs represent a significant step forward in merging continuous-depth models with Bayesian inference. While they offer superior calibration and robustness, challenges remain regarding computational cost and the integration of standard techniques like Batch Normalization, which currently conflict with the Lipschitz continuity requirements of SDE solvers. Future research will focus on regularizing SDEs for faster solving and scaling these methods to larger architectures like Transformers.
Summary updated