BNNpriors: A library for Bayesian neural network inference with different prior distributions

Vincent Fortuin, Adrià Garriga-Alonso, Mark van der Wilk, Laurence Aitchison

Abstract

Bayesian neural networks (BNNs) offer a principled framework for uncertainty estimation, yet their practical performance often hinges on the choice of weight priors. While isotropic Gaussian priors are the standard default, they frequently fail to capture the complex structure of neural network weights, leading to the "cold posterior effect" where models perform better with artificially sharpened distributions. We introduce BNNpriors, a PyTorch-based library that enables efficient Markov Chain Monte Carlo (MCMC) inference across a broad spectrum of priors, including heavy-tailed, hierarchical, and mixture distributions. Our results demonstrate that sophisticated prior selection can significantly improve model calibration and predictive performance, potentially resolving fundamental misspecification issues in BNNs.

Introduction: The Challenge of Prior Selection

In Bayesian deep learning, we seek to infer a posterior distribution over model weights rather than a single point estimate. This approach is theoretically ideal for tasks requiring calibrated uncertainty. However, the common practice of using independent Gaussian priors on weights often leads to suboptimal results. A notable symptom of this is the "cold posterior effect," where researchers find that scaling the log-posterior by an inverse temperature $1/T$ (where $T < 1$) improves performance. This suggests that the standard Bayesian model is misspecified—the prior does not accurately reflect our true beliefs about the weight distribution.

BNNpriors was developed to address this gap by providing a modular, scalable framework for experimenting with non-Gaussian priors. By moving beyond simple bell curves, researchers can better align their models with the underlying data and architectural inductive biases.

Methods and Implementation

BNNpriors is built as an extension of the PyTorch ecosystem. It replaces standard torch.Parameter objects with a custom Prior class, allowing for seamless integration into existing neural network architectures.

Diverse Prior Support

The library supports a wide array of distributions out-of-the-box. These range from simple heavy-tailed distributions like Laplace and Student-t to more complex hierarchical and mixture models.

Mathematical densities of the various prior distributions supported by the BNNpriors library.

Advanced MCMC Inference

To handle the high-dimensional and non-convex landscapes of neural networks, BNNpriors implements state-of-the-art MCMC schemes, including Stochastic Gradient Langevin Dynamics (SGLD) and Hamiltonian Monte Carlo (HMC). The authors specifically recommend the VerletSGLDReject approach, a "gradient-guided Monte Carlo" method that provides accurate, scalable inference even with stochastic gradients.

The library also includes diagnostic tools, such as "kinetic temperature" estimation, to ensure that MCMC chains have converged and that the chosen learning rates are appropriate for the target posterior temperature.

Experimental Results

The utility of BNNpriors is demonstrated through extensive benchmarking on standard computer vision datasets: MNIST, FashionMNIST, and CIFAR-10. The primary evaluation tool is the "tempering curve," which plots performance metrics against the posterior temperature $T$.

Performance metrics (Error, NLL, ECE, OOD AUROC) across different temperatures for various priors on MNIST, FashionMNIST, and CIFAR-10.

As shown in the results, non-Gaussian priors consistently outperform the standard Gaussian baseline across several dimensions:

  1. Predictive Accuracy: Heavy-tailed and correlated priors often achieve lower error rates.
  2. Likelihood and Calibration: Models using these priors show better Negative Log-Likelihood (NLL) and lower Expected Calibration Error (ECE), meaning their confidence levels are more trustworthy.
  3. Out-of-Distribution (OOD) Detection: The ability to identify data from outside the training distribution is significantly enhanced with better prior choices.

Crucially, the tempering curves reveal that for certain priors, the optimal temperature is much closer to $T=1$ (the true Bayesian posterior) than for Gaussian priors. This provides strong evidence that the "cold posterior effect" is a consequence of poor prior choice.

Key Insights

  • Architecture-Specific Priors: The authors observed that heavy-tailed priors (Student-t, Cauchy) are particularly effective for fully-connected networks, while correlated priors provide superior results for Convolutional Neural Networks (CNNs).
  • Hierarchical Modeling: The modular design allows for hierarchical priors, where the hyperparameters of a weight prior (e.g., the scale of a Laplace distribution) are themselves treated as random variables.
  • Scalability: By utilizing cyclical learning rates and preconditioning, BNNpriors demonstrates that high-quality Bayesian inference is possible even for deep, modern architectures without prohibitive computational costs.

Limitations and Future Directions

Despite its capabilities, BNNpriors has areas for further development. Currently, hierarchical priors require joint inference of weights and hyperparameters; the addition of Gibbs sampling or reversible jump MCMC could provide more flexibility. Furthermore, while the library excels in vision tasks, its application to Recurrent Neural Networks (RNNs) and Transformers is an ongoing area of research. Finally, automating the selection of the "best" prior through marginal likelihood estimation remains a significant challenge that the authors intend to tackle in future iterations.

Summary updated

Abstract

Bayesian neural networks have shown great promise in many applications where calibrated uncertainty estimates are crucial and can often also lead to a higher predictive performance. However, it remains challenging to choose a good prior distribution over their weights. While isotropic Gaussian priors are often chosen in practice due to their simplicity, they do not reflect our true prior beliefs well and can lead to suboptimal performance. Our new library, BNNpriors, enables state-of-the-art Markov Chain Monte Carlo inference on Bayesian neural networks with a wide range of predefined priors, including heavy-tailed ones, hierarchical ones, and mixture priors. Moreover, it follows a modular approach that eases the design and implementation of new custom priors. It has facilitated foundational discoveries on the nature of the cold posterior effect in Bayesian neural networks and will hopefully catalyze future research as well as practical applications in this area.
BibTeX
@article{fortuin2021bnnpriors,
  author = {Vincent Fortuin and Adrià Garriga-Alonso and Mark van der Wilk and Laurence Aitchison},
  title = {{BNNpriors: A library for Bayesian neural network inference with different prior distributions}},
  journal = {Softw. Impacts},
  year = {2021},
  doi = {10.1016/j.simpa.2021.100079},
}
Figure