Scaling Hamiltonian Monte Carlo inference for Bayesian neural networks with symmetric splitting

Adam D. Cobb, Brian Jalaian

Abstract

Hamiltonian Monte Carlo (HMC) is the gold standard for Bayesian inference in high-dimensional models, yet its requirement for full-dataset gradients hinders scalability. This paper introduces a Symmetric Split Leapfrog integration scheme that enables HMC to scale to large datasets without the bias of stochastic gradients. By maintaining mathematical symmetry and reversibility, the method preserves the Metropolis-Hastings step, ensuring convergence to the true posterior. Experiments on benchmarks like Fashion MNIST and the ACIDS vehicle classification task demonstrate superior predictive accuracy and significantly improved uncertainty calibration compared to stochastic MCMC alternatives.

Introduction

Bayesian Neural Networks (BNNs) offer a robust framework for quantifying uncertainty, which is critical for safety-sensitive applications. While HMC provides efficient exploration of parameter spaces, its computational cost traditionally scales linearly with dataset size. Stochastic Gradient MCMC (SG-MCMC) methods like SGLD and SGHMC address this by using mini-batches but sacrifice the Metropolis-Hastings (MH) correction, leading to biased samples and overconfident predictions. This work seeks to retain the rigor of full HMC while gaining the memory efficiency of batch processing.

Methodology: Symmetric Splitting

The proposed Symmetric Split HMC partitions the potential energy (data likelihood) into $M$ batches. Unlike previous splitting methods that rely on randomization, this approach utilizes a strictly symmetric sequence of partial momentum and parameter updates. This symmetry ensures the integrator is reversible, allowing for the inclusion of an MH accept/reject step. This "Full HMC" behavior is maintained while processing data in memory-efficient chunks, making it suitable for single-GPU training of deep architectures.

Figure 1: 1D Synthetic Regression demonstrating the model's ability to capture epistemic and aleatoric uncertainty with a high acceptance rate (95%).

Experiments and Results

The method was evaluated on several tasks, including 1D regression, Fashion MNIST (FMNIST), and the ACIDS vehicle classification dataset.

  • Performance: On the real-world ACIDS dataset, the proposed method achieved a state-of-the-art accuracy of 84.4% and the lowest negative log-likelihood (0.51), outperforming SGD, SGLD, and SGHMC.
  • Acceptance Rates: In the FMNIST task, Symmetric Split HMC achieved an acceptance rate of 0.89, significantly higher than naive (0.68) or randomized (0.66) splitting variants.
  • Memory Efficiency: The authors demonstrated that by splitting data into subsets, they could train on datasets that would otherwise exceed the memory limits of a single consumer GPU (e.g., RTX 2080).

Uncertainty Calibration

A key advantage of the symmetric splitting approach is the quality of its uncertainty estimates. Analysis of the posterior predictive entropy on misclassified data reveals that Symmetric Split HMC is significantly less overconfident than SGD or SG-MCMC methods. As shown in the empirical CDF below, the novel method assigns higher entropy (uncertainty) to points it misclassifies, a vital trait for safety-critical AI.

Figure 2: Empirical CDF of Posterior Predictive Entropy over misclassified data. The Novel Sym. Split HMC shows a distribution shifted toward higher entropy compared to SGD and SG-MCMC methods.

Conclusion

Symmetric Split HMC bridges the gap between the rigorous inference of full HMC and the scalability of batch-based methods. While computationally more demanding than stochastic gradients in terms of wall-clock time, its ability to provide well-calibrated uncertainty on a single GPU makes it a compelling choice for high-stakes applications. Future work includes exploring Riemannian HMC extensions to further improve sampling efficiency in complex geometries.

Summary updated

Abstract

Hamiltonian Monte Carlo (HMC) is a Markov chain Monte Carlo (MCMC) approach that exhibits favourable exploration properties in high-dimensional models such as neural networks. Unfortunately, HMC ha...
BibTeX
@inproceedings{cobb2021scaling,
  author = {Adam D. Cobb and Brian Jalaian},
  title = {{Scaling Hamiltonian Monte Carlo inference for Bayesian neural networks with symmetric splitting}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=4ZWeXKDS_I},
}
Figure