Abstract
Hamiltonian Monte Carlo (HMC) is the gold standard for Bayesian inference in high-dimensional models, yet its requirement for full-dataset gradients hinders scalability. This paper introduces a Symmetric Split Leapfrog integration scheme that enables HMC to scale to large datasets without the bias of stochastic gradients. By maintaining mathematical symmetry and reversibility, the method preserves the Metropolis-Hastings step, ensuring convergence to the true posterior. Experiments on benchmarks like Fashion MNIST and the ACIDS vehicle classification task demonstrate superior predictive accuracy and significantly improved uncertainty calibration compared to stochastic MCMC alternatives.
Introduction
Bayesian Neural Networks (BNNs) offer a robust framework for quantifying uncertainty, which is critical for safety-sensitive applications. While HMC provides efficient exploration of parameter spaces, its computational cost traditionally scales linearly with dataset size. Stochastic Gradient MCMC (SG-MCMC) methods like SGLD and SGHMC address this by using mini-batches but sacrifice the Metropolis-Hastings (MH) correction, leading to biased samples and overconfident predictions. This work seeks to retain the rigor of full HMC while gaining the memory efficiency of batch processing.
Methodology: Symmetric Splitting
The proposed Symmetric Split HMC partitions the potential energy (data likelihood) into $M$ batches. Unlike previous splitting methods that rely on randomization, this approach utilizes a strictly symmetric sequence of partial momentum and parameter updates. This symmetry ensures the integrator is reversible, allowing for the inclusion of an MH accept/reject step. This "Full HMC" behavior is maintained while processing data in memory-efficient chunks, making it suitable for single-GPU training of deep architectures.
Experiments and Results
The method was evaluated on several tasks, including 1D regression, Fashion MNIST (FMNIST), and the ACIDS vehicle classification dataset.
- Performance: On the real-world ACIDS dataset, the proposed method achieved a state-of-the-art accuracy of 84.4% and the lowest negative log-likelihood (0.51), outperforming SGD, SGLD, and SGHMC.
- Acceptance Rates: In the FMNIST task, Symmetric Split HMC achieved an acceptance rate of 0.89, significantly higher than naive (0.68) or randomized (0.66) splitting variants.
- Memory Efficiency: The authors demonstrated that by splitting data into subsets, they could train on datasets that would otherwise exceed the memory limits of a single consumer GPU (e.g., RTX 2080).
Uncertainty Calibration
A key advantage of the symmetric splitting approach is the quality of its uncertainty estimates. Analysis of the posterior predictive entropy on misclassified data reveals that Symmetric Split HMC is significantly less overconfident than SGD or SG-MCMC methods. As shown in the empirical CDF below, the novel method assigns higher entropy (uncertainty) to points it misclassifies, a vital trait for safety-critical AI.
Conclusion
Symmetric Split HMC bridges the gap between the rigorous inference of full HMC and the scalability of batch-based methods. While computationally more demanding than stochastic gradients in terms of wall-clock time, its ability to provide well-calibrated uncertainty on a single GPU makes it a compelling choice for high-stakes applications. Future work includes exploring Riemannian HMC extensions to further improve sampling efficiency in complex geometries.
Summary updated