Abstract
Hamiltonian Monte Carlo (HMC) is a Markov chain Monte Carlo (MCMC) approach that exhibits favourable exploration properties in high-dimensional models such as neural networks. Unfortunately, HMC has limited use in large-data regimes and little work has explored suitable approaches that aim to preserve the entire Hamiltonian. In our work, we introduce a new symmetric integration scheme for split HMC that does not rely on stochastic gradients. We show that our new formulation is more efficient than previous approaches and is easy to implement with a single GPU. As a result, we are able to perform full HMC over common deep learning architectures using entire data sets. In addition, when we compare with stochastic gradient MCMC, we show that our method achieves better performance in both accuracy and uncertainty quantification. Our approach demonstrates HMC as a feasible option when considering inference schemes for large-scale machine learning problems.
Introduction
In the domain of Bayesian Neural Networks (BNNs), Hamiltonian Monte Carlo (HMC) is frequently cited as the "gold standard" for its ability to explore high-dimensional parameter spaces effectively. However, its primary limitation is scalability; standard HMC requires computing the gradient of the likelihood for the entire dataset at every simulation step. To mitigate this, researchers often employ Stochastic Gradient MCMC (SGMCMC), which utilizes mini-batches. While computationally efficient, SGMCMC introduces noise and bias, frequently resulting in overconfident models with poor uncertainty representation. This paper proposes a middle ground: a data-splitting approach that preserves the formal properties of HMC, such as reversibility and the Metropolis-Hastings step, ensuring samples are drawn from the true posterior distribution.
Methodology: Novel Symmetric Split HMC
The proposed approach introduces a Novel Symmetric Split HMC scheme. The core innovation involves partitioning the Hamiltonian—the system's energy function—into $M$ components based on data subsets.
Symmetric Integration
Unlike previous randomized splitting methods that require data shuffling to maintain validity, this method employs a fixed, symmetric sequence of updates. The integrator updates momentum, cycles through data subsets to update parameters, and then cycles back in reverse order. This perfect symmetry ensures mathematical reversibility, a prerequisite for MCMC convergence.
Kinetic Energy Rescaling
A critical technical contribution is the rescaling of the kinetic energy term by a factor $D$, related to the number of splits. This adjustment maintains simulation stability and accuracy despite data partitioning. The method is optimized for GPU execution and is integrated into the hamiltorch Python package.
Experimental Evaluation
The efficacy of the symmetric split HMC was evaluated across four distinct scenarios:
- 1D Regression: A synthetic task for visualizing epistemic uncertainty.
- Fashion MNIST (FMNIST): A benchmark classification task (48k images) split into three subsets.
- CIFAR-10: A subset of 1,000 images used to evaluate the impact of a higher number of splits (10 subsets).
- ACIDS (Acoustic-seismic Classification Identification Data Set): A real-world dataset of 223 audio recordings for vehicle classification, characterized by high dimensionality and class imbalance.
Results and Discussion
The experimental results highlight several advantages of the symmetric splitting scheme regarding efficiency and predictive quality.
Acceptance Rates and Exploration
A primary metric for MCMC efficiency is the Metropolis-Hastings acceptance rate. The novel symmetric split achieves significantly higher rates (often exceeding 80%) compared to full HMC or naive splitting. A higher acceptance rate indicates that the chain explores the parameter space more freely and efficiently.
Uncertainty Quantification
In regression tasks, the symmetric split HMC produces wider, more realistic uncertainty estimates in regions with sparse data, avoiding the overconfidence typical of stochastic methods. On the ACIDS task, the model's Mutual Information effectively distinguished between correct and incorrect predictions, a "near-separability" property crucial for safety-critical applications.
Performance Metrics
- Accuracy: On the ACIDS task, the Novel Symmetric Split (NSS) HMC achieved 84.4% accuracy, surpassing SGHMC (82.6%) and SGLD (78.6%).
- Calibration: NSS HMC yielded the lowest Brier Score (0.228), indicating superior probability calibration.
- Efficiency: Splitting enabled HMC on datasets that would otherwise exceed single-GPU memory limits, such as a theoretical 3x increase in training data for FMNIST.
Conclusion
While the symmetric split HMC demonstrates significant improvements, challenges remain regarding mass matrix tuning and hyperparameter sensitivity. Future work may explore geometrically-inspired metrics to further enhance performance. Nonetheless, this approach establishes HMC as a viable and robust inference scheme for large-scale machine learning problems.
Summary updated