Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry (Extended Abstract)

Jonas Gregor Wiese, Lisa Wimmer, Theodore Papamarkou, Bernd Bischl, Stephan Günnemann, David RügamerIJCAI 2024

Introduction

Bayesian Neural Networks (BNNs) provide a robust framework for uncertainty quantification by treating model weights as probability distributions. However, the resulting posterior distributions are notoriously difficult to sample due to their high dimensionality and extreme multi-modality. While Markov Chain Monte Carlo (MCMC) is theoretically the gold standard, it is often dismissed as computationally prohibitive for deep learning. This paper challenges that notion by demonstrating that much of the posterior's complexity is redundant symmetry.

The Challenge of Parameter Redundancy

The parameter landscape of a BNN is filled with "Equioutput Transformations"—different weight configurations that yield identical output functions. These symmetries arise from neuron permutations in hidden layers and sign-flip symmetries in activation functions like tanh.

Exponential growth of parameter redundancy with network size.

As shown in Figure 1, the number of symmetric copies of a single mode grows exponentially with the number of neurons. For a simple network with 128 neurons, the redundancy factor reaches approximately $1.31 \times 10^{254}$. This astronomical redundancy implies that the functional space is significantly smaller than the parameter space. If two chains find different parameters that describe the same function, they have not added new information to the uncertainty quantification.

Methodology: Exploiting Functional Diversity

The authors propose focusing on "functionally unique" modes rather than exhaustive parameter exploration. They prove that the posterior predictive density (PPD) can be accurately estimated by integrating over a small "reference set" of unique parameters.

Instead of attempting to navigate the entire landscape with a single long MCMC chain—which often becomes trapped in local modes—the authors derive a probabilistic upper bound for the number of independent, parallelizable chains required to visit every functionally diverse mode. By running a specific number of shorter chains from random starting points, the method efficiently captures the necessary functional diversity for state-of-the-art predictive performance.

Experimental Evaluation

The approach was evaluated on synthetic datasets and a suite of UCI regression benchmarks. The results demonstrate that the proposed method consistently outperforms or matches standard single-chain MCMC and Deep Ensembles (DE).

Comparison of Mean Log Pointwise Predictive Density (LPPD) across datasets.

As detailed in Figure 6, the method shows significant gains in Mean Log Pointwise Predictive Density (LPPD). For instance, on the "Yacht" dataset, the proposed MCMC approach achieved an LPPD of 3.31, vastly superior to the 0.15 achieved by single-chain MCMC and the -2.18 of Deep Ensembles. Furthermore, the functional diversity was found to saturate relatively quickly (around 1,200 samples), suggesting that the PPD converges long before the parameter space is fully explored.

Conclusion and Future Work

By exploiting architectural symmetries, this work provides a theoretical and practical path toward efficient MCMC in BNNs. While the current analysis focuses on Multi-Layer Perceptrons (MLPs) and requires some assumptions about mode distribution, it highlights a shift in focus from parameter-space coverage to functional-space coverage. Future research will aim to extend these symmetry-aware sampling techniques to more complex architectures like CNNs and Transformers.

Summary updated

BibTeX
@inproceedings{wiese2024towards,
  author = {Jonas Gregor Wiese and Lisa Wimmer and Theodore Papamarkou and Bernd Bischl and Stephan Günnemann and David Rügamer},
  title = {{Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure