Introduction
Bayesian Neural Networks (BNNs) offer a principled framework for uncertainty quantification in deep learning. However, their practical application is hindered by the immense difficulty of sampling from the posterior distribution of weights. As networks grow wider—a trend central to modern deep learning—the parameter space expands, leading to the "curse of dimensionality." Standard Markov Chain Monte Carlo (MCMC) methods, such as the Metropolis-Adjusted Langevin Algorithm (LMC), typically require infinitesimally small step sizes to maintain reasonable acceptance rates in high-dimensional spaces, resulting in prohibitively slow exploration.
This research demonstrates that specific MCMC algorithms, namely the preconditioned Crank-Nicolson (pCN) and its Langevin variant (pCNL), effectively overcome these limitations. The authors prove that for wide BNNs, these samplers actually become more efficient as the network width increases, turning the curse of dimensionality into a "blessing of width."
Methodology
The core innovation lies in leveraging the convergence of BNN posteriors to Gaussian processes in the infinite-width limit. By employing a specific reparameterization and utilizing dimension-robust samplers, the authors ensure stable performance regardless of the number of parameters.
Reparameterization and Samplers
The approach utilizes a data-dependent transformation $\phi = T^{-1}(\theta)$ for the readout layer, mapping the complex posterior into a space that behaves more like a standard Gaussian. Two primary samplers are evaluated:
- pCN (Preconditioned Crank-Nicolson): Designed for infinite-dimensional Hilbert spaces, pCN uses a proposal mechanism that preserves the Gaussian prior, making it inherently robust to increasing dimensions.
- pCNL (pCN-Langevin): This variant incorporates gradient information to guide the sampler toward regions of higher probability.
The theoretical foundation of this work is the proof that for any fixed step size, the acceptance rate of pCN and pCNL converges to 1 as the network width $d \to \infty$.
Experimental Results
The theoretical findings were validated using the CIFAR-10 image classification benchmark. The experiments focused on one-layer fully connected networks with widths ranging from 128 to 8192.
Acceptance Rate Dynamics
A critical metric for MCMC efficiency is the acceptance rate. As shown below, standard LMC performance degrades rapidly as the network dimension increases, especially at larger step sizes. In contrast, pCN and pCNL maintain or even improve their acceptance rates as the width grows.
Sampling Efficiency
The practical benefit of these algorithms is best captured by the Effective Sample Size (ESS) per step. ESS measures the number of independent samples a chain effectively produces.
As illustrated above, the ESS for LMC plummets as the network widens. Conversely, pCN and pCNL show a steady increase in ESS, confirming that wider networks allow these specialized samplers to explore the posterior more effectively. Furthermore, convergence diagnostics using the Gelman-Rubin statistic ($\hat{R}$) indicated that pCN and pCNL chains reached stationarity significantly faster than LMC in wide configurations.
Discussion and Conclusion
The results highlight a counter-intuitive phenomenon: increasing the number of parameters in a BNN can simplify the sampling problem if the right algorithms are used. As the network becomes wider, the posterior distribution becomes more Gaussian-like, which perfectly suits the design of pCN-based samplers.
While pCNL provides gradient guidance, the authors noted that its relative advantage over the simpler pCN diminishes in the very wide limit. This suggests that for extremely large-scale models, the computationally cheaper pCN might offer the best balance of performance and efficiency. Future work will involve extending these findings to deeper architectures and more complex convolutional layers.
Summary updated