Function-Space MCMC for Bayesian Wide Neural Networks

Lucia Pezzetti, Stefano Favaro, Stefano PeluchettiAISTATS 2025 · OpenReview

Introduction

Bayesian Neural Networks (BNNs) offer a principled framework for uncertainty quantification in deep learning. However, their practical application is hindered by the immense difficulty of sampling from the posterior distribution of weights. As networks grow wider—a trend central to modern deep learning—the parameter space expands, leading to the "curse of dimensionality." Standard Markov Chain Monte Carlo (MCMC) methods, such as the Metropolis-Adjusted Langevin Algorithm (LMC), typically require infinitesimally small step sizes to maintain reasonable acceptance rates in high-dimensional spaces, resulting in prohibitively slow exploration.

This research demonstrates that specific MCMC algorithms, namely the preconditioned Crank-Nicolson (pCN) and its Langevin variant (pCNL), effectively overcome these limitations. The authors prove that for wide BNNs, these samplers actually become more efficient as the network width increases, turning the curse of dimensionality into a "blessing of width."

Methodology

The core innovation lies in leveraging the convergence of BNN posteriors to Gaussian processes in the infinite-width limit. By employing a specific reparameterization and utilizing dimension-robust samplers, the authors ensure stable performance regardless of the number of parameters.

Reparameterization and Samplers

The approach utilizes a data-dependent transformation $\phi = T^{-1}(\theta)$ for the readout layer, mapping the complex posterior into a space that behaves more like a standard Gaussian. Two primary samplers are evaluated:

  1. pCN (Preconditioned Crank-Nicolson): Designed for infinite-dimensional Hilbert spaces, pCN uses a proposal mechanism that preserves the Gaussian prior, making it inherently robust to increasing dimensions.
  2. pCNL (pCN-Langevin): This variant incorporates gradient information to guide the sampler toward regions of higher probability.

The theoretical foundation of this work is the proof that for any fixed step size, the acceptance rate of pCN and pCNL converges to 1 as the network width $d \to \infty$.

Experimental Results

The theoretical findings were validated using the CIFAR-10 image classification benchmark. The experiments focused on one-layer fully connected networks with widths ranging from 128 to 8192.

Acceptance Rate Dynamics

A critical metric for MCMC efficiency is the acceptance rate. As shown below, standard LMC performance degrades rapidly as the network dimension increases, especially at larger step sizes. In contrast, pCN and pCNL maintain or even improve their acceptance rates as the width grows.

Acceptance rates of pCN, pCNL, and LMC across different dimensions and step sizes.

Sampling Efficiency

The practical benefit of these algorithms is best captured by the Effective Sample Size (ESS) per step. ESS measures the number of independent samples a chain effectively produces.

Effective Sample Size (ESS) per step for weights as a function of network width.

As illustrated above, the ESS for LMC plummets as the network widens. Conversely, pCN and pCNL show a steady increase in ESS, confirming that wider networks allow these specialized samplers to explore the posterior more effectively. Furthermore, convergence diagnostics using the Gelman-Rubin statistic ($\hat{R}$) indicated that pCN and pCNL chains reached stationarity significantly faster than LMC in wide configurations.

Discussion and Conclusion

The results highlight a counter-intuitive phenomenon: increasing the number of parameters in a BNN can simplify the sampling problem if the right algorithms are used. As the network becomes wider, the posterior distribution becomes more Gaussian-like, which perfectly suits the design of pCN-based samplers.

While pCNL provides gradient guidance, the authors noted that its relative advantage over the simpler pCN diminishes in the very wide limit. This suggests that for extremely large-scale models, the computationally cheaper pCN might offer the best balance of performance and efficiency. Future work will involve extending these findings to deeper architectures and more complex convolutional layers.

Summary updated

Abstract

Bayesian Neural Networks represent a fascinating confluence of deep learning and probabilistic reasoning, offering a compelling framework for understanding uncertainty in complex predictive models. In this paper, we investigate the use of the preconditioned Crank-Nicolson algorithm and its Langevin version to sample from a reparametrised posterior distribution of the neural network's weights, as the widths grow larger. In addition to being robust in the infinite-dimensional setting, we prove that the acceptance probabilities of the proposed algorithms approach 1 as the width of the network increases, independently of any stepsize tuning. Moreover, we examine and compare how the mixing speeds of the underdamped Langevin Monte Carlo, the preconditioned Crank-Nicolson and the preconditioned Crank-Nicolson Langevin samplers are influenced by changes in the network width in some real-world cases. Our findings suggest that, in wide Bayesian Neural Networks configurations, the preconditioned Crank-Nicolson algorithm allows for a scalable and more efficient sampling of the reparametrised posterior distribution, as also evidenced by a higher effective sample size and improved diagnostic results compared with the other analysed algorithms.
BibTeX
@inproceedings{pezzetti2025function,
  author = {Lucia Pezzetti and Stefano Favaro and Stefano Peluchetti},
  title = {{Function-Space MCMC for Bayesian Wide Neural Networks}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=Pgcc6fI0dw},
}
Figure