Sampling from Bayesian Neural Network Posteriors with Symmetric Minibatch Splitting Langevin Dynamics

Daniel Paulin, Peter A. Whalley, Neil K. Chada, Benedict J. LeimkuhlerAISTATS 2025

Abstract

This paper introduces Symmetric Minibatch Splitting-UBU (SMS-UBU), a scalable Langevin dynamics algorithm designed to address the sampling challenges inherent in Bayesian Neural Networks (BNNs). By utilizing a novel symmetric minibatch ordering—processing data in a forward-then-backward sequence—the method achieves second-order numerical accuracy. This approach significantly reduces the sampling bias typically introduced by stochastic gradients in large-scale datasets. Experimental evaluations on computer vision benchmarks demonstrate that SMS-UBU provides superior uncertainty calibration and maintains high predictive performance even with larger integration step sizes.

Introduction

Bayesian Neural Networks (BNNs) are highly valued for their ability to provide robust uncertainty quantification. However, their application to large-scale datasets is hindered by the computational difficulty of sampling from complex posterior distributions. Standard Markov Chain Monte Carlo (MCMC) methods, such as Langevin dynamics, often rely on stochastic gradients calculated from minibatches to remain scalable. Unfortunately, this stochasticity introduces significant sampling bias, often forcing a trade-off where algorithms drop from second-order to half-order accuracy to manage noise. SMS-UBU aims to resolve this by restoring high-order accuracy through structural symmetry in data processing.

Methodology: Symmetric Minibatch Splitting

The core innovation of this work is the Symmetric Minibatch Splitting (SMS) scheme applied to the UBU (Update-Boost-Update) integrator within a kinetic Langevin dynamics framework.

Unlike standard stochastic gradient methods that select minibatches randomly at each step, SMS-UBU processes a fixed set of minibatches in a specific order: $1, 2, \dots, N_m$, followed immediately by the reverse sequence $N_m, N_{m-1}, \dots, 1$. This symmetric arrangement ensures that the leading-order errors from numerical integration cancel out. Mathematically, this preserves second-order accuracy $O(h^2)$, where $h$ is the step size, even in the presence of stochastic gradients. The authors prove that this method maintains a sampling bias of $O(h^2 d^{1/2})$, offering a substantial theoretical improvement over traditional stochastic gradient MCMC methods.

Figure 4: Wasserstein bias across different integration methods (UBU, BAOAB, EM). The SMS variants (black lines) consistently exhibit lower bias and better scaling compared to standard stochastic gradient versions (red lines).

Experimental Results

The researchers evaluated SMS-UBU against several baselines, including SG-HMC and Stochastic Weight Averaging (SWA), across benchmarks such as Fashion-MNIST, Celeb-A, and Chest X-ray datasets.

Bias and Stability

Theoretical gains in bias control were empirically validated. As shown in the Wasserstein distance analysis, the SMS-based methods maintain significantly lower bias across various step sizes compared to their non-symmetric counterparts. Furthermore, the method demonstrates remarkable robustness to the choice of step size. While traditional methods like SG-HMC experience a sharp decline in accuracy as the step size increases, SMS-UBU remains stable and accurate.

Figure 5: Test set accuracy as a function of step size. While SG-HMC (green) shows significant performance degradation at larger step sizes, SMS-UBU (purple) maintains high, stable accuracy.

Uncertainty Calibration

Beyond raw accuracy, SMS-UBU excels in uncertainty quantification. Across all tested datasets, it consistently outperformed standard training and SWA in terms of calibration metrics, including Negative Log-Likelihood (NLL), Adaptive Calibration Error (ACE), and Ranked Probability Scores (RPS). In CIFAR-10 experiments, the BNN approach achieved an accuracy of 89.79%, outperforming more computationally intensive Hamiltonian Monte Carlo (HMC) methods that required significantly more training epochs.

Conclusion and Future Work

SMS-UBU provides a mathematically grounded and computationally efficient way to sample from BNN posteriors. By enforcing symmetry in minibatch processing, it restores second-order accuracy to scalable MCMC. While currently focused on localized sampling near high-quality solutions, future work may explore combining this scheme with mode-connectivity frameworks to capture broader functional diversity across the neural network landscape.

Summary updated

BibTeX
@inproceedings{paulin2025sampling,
  author = {Daniel Paulin and Peter A. Whalley and Neil K. Chada and Benedict J. Leimkuhler},
  title = {{Sampling from Bayesian Neural Network Posteriors with Symmetric Minibatch Splitting Langevin Dynamics}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
}
Figure