Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient

Vu C. Dinh, Lam Si Tung Ho, Cuong V. Nguyen

Abstract: While Hamiltonian Monte Carlo (HMC) is often considered the "gold standard" for Bayesian Deep Learning, its performance on non-smooth architectures like ReLU neural networks is significantly compromised. This paper, presented at NeurIPS 2024, demonstrates that the non-differentiability of ReLU activations causes the leapfrog integrator to lose its high-order accuracy, leading to higher rejection rates and requiring much smaller step sizes as model dimensions grow. We provide theoretical proofs and empirical evidence showing that smooth activation functions like Sigmoid are substantially more efficient for HMC-based inference.

Introduction

Bayesian Neural Networks (BNNs) provide a robust framework for quantifying uncertainty in deep learning. To explore the complex posterior distributions of these models, practitioners frequently turn to Hamiltonian Monte Carlo (HMC). HMC relies on a "leapfrog integrator"—a numerical solver that simulates physical dynamics to explore parameter space efficiently.

In smooth mathematical settings, such as those using Sigmoid activations, this integrator is highly accurate, maintaining a local error rate of $O(\epsilon^3)$. However, the ubiquitous Rectified Linear Unit (ReLU) activation function introduces a sharp "kink" at zero where it is non-differentiable. While it is often assumed that the sampler rarely hits these points exactly, this research proves that simply crossing these boundaries during a simulation step creates a massive error jump, sabotaging the algorithm's efficiency.

The Problem of Non-Differentiability

The core issue lies in the "Kink Penalty." Every time a neuron switches between active and inactive states during a leapfrog step, the Hamiltonian (the total energy the algorithm seeks to conserve) incurs a significant error. In a neural network with thousands or millions of neurons, these errors accumulate rapidly.

Theoretical analysis reveals that when a leapfrog step crosses a surface of non-differentiability, the local error jumps from the standard $O(\epsilon^3)$ to $\Omega(\epsilon)$. Consequently, to maintain stability and a reasonable acceptance rate, the step size $\epsilon$ must scale more restrictively with the dimension $d$—specifically as $d^{-1/2}$ for ReLU-like densities, compared to the $d^{-1/4}$ scaling typical for smooth functions.

Methodology

The authors validated these findings through a combination of theoretical derivations and comparative simulations:

  • Theoretical Error Analysis: Deriving the Hamiltonian error jump for non-smooth densities.
  • Optimal Tuning Theory: Adapting classical HMC scaling limits to piece-wise affine densities, finding that the optimal acceptance rate for ReLU networks is approximately 0.45, lower than the standard 0.651 for smooth targets.
  • Comparative Testing: Evaluating Sigmoid, ReLU, and Leaky ReLU architectures on a 1D synthetic regression task and the high-dimensional UTKFace image regression dataset.

Key Results

The empirical results confirm the theoretical predictions regarding efficiency and stability. As shown in the figures below, Sigmoid-based networks maintain high and stable acceptance rates across varying simulation lengths and step sizes. In contrast, ReLU and Leaky ReLU networks exhibit a sharp decay in performance.

Comparison of HMC acceptance rates across activation functions. The left plot shows the decay in acceptance rate as the number of leapfrog steps (L) increases for ReLU-based networks. The right plot illustrates the sensitivity of non-smooth networks to the step size (ε).

On the real-world UTKFace dataset, the Sigmoid network achieved a significantly higher average acceptance rate (0.54) compared to ReLU (0.22) and Leaky ReLU (0.17). Furthermore, the Sigmoid network achieved lower Mean Squared Error (MSE), suggesting that the improved sampling efficiency translates directly into better predictive performance.

Conclusion

The inefficiency of HMC on ReLU networks is not a matter of mathematical "correctness"—the sampler still targets the right distribution—but rather a matter of computational feasibility. For practitioners utilizing HMC for Bayesian inference, smooth activation functions like Sigmoid or Softplus offer a clear advantage, often improving efficiency by a factor of 4x or more. Future work will explore whether specialized numerical solvers or alternative integrators can mitigate the penalty imposed by non-differentiable activation functions in even deeper or more complex architectures like CNNs.

Summary updated

Abstract

We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of $\Omega(\epsilon)$ rather than the classical error rate of $\mathcal{O}(\epsilon^3)$. This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks.
BibTeX
@inproceedings{dinh2024hamiltonian,
  author = {Vu C. Dinh and Lam Si Tung Ho and Cuong V. Nguyen},
  title = {{Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=aBmiyi7iA7},
}
Figure