BNNpriors: A library for Bayesian neural network inference with different prior distributions

Vincent Fortuin, Adrià Garriga-Alonso, Mark van der Wilk, Laurence Aitchison

TL;DR

BNNpriors is a new open-source PyTorch library designed to move Bayesian Neural Networks (BNNs) beyond simple Gaussian weight priors. By enabling efficient MCMC inference with heavy-tailed, hierarchical, and mixture distributions, it helps researchers improve model calibration and solve the mysterious "cold posterior effect."

What Problem Does This Paper Address?

In theory, Bayesian Neural Networks (BNNs) are the gold standard for uncertainty estimation. Instead of finding a single "best" set of weights, they infer a probability distribution over all possible weights. However, in practice, most researchers default to isotropic Gaussian priors—essentially assuming every weight follows a simple, independent bell curve centered at zero.

The problem? These simple priors rarely reflect the complex reality of neural network weights. This mismatch leads to the "cold posterior effect," a phenomenon where BNNs perform better when their posterior distributions are artificially "cooled" (sharpened). This suggests that our standard BNN models are fundamentally misspecified. The authors created BNNpriors to provide the community with the tools to experiment with more sophisticated priors that might actually match the data.

Methods

The library is built on top of PyTorch, making it familiar to most deep learning practitioners. It replaces standard torch.Parameter objects with a custom Prior class.

Key technical features include:

  • Diverse Prior Support: Out-of-the-box support for Laplace, Student-t, Cauchy, Multivariate Gaussian/t, Hierarchical, and Mixture priors.
  • Advanced Inference: It implements state-of-the-art Markov Chain Monte Carlo (MCMC) schemes, including Stochastic Gradient Langevin Dynamics (SGLD) and Hamiltonian Monte Carlo (HMC).
  • VerletSGLDReject: The authors specifically recommend this "gradient-guided Monte Carlo" approach, which provides accurate, scalable inference even with stochastic gradients.
  • Diagnostics: The library includes tools to estimate "kinetic temperature," allowing users to verify if their MCMC chains have actually converged or if the learning rate is too high.

Datasets

The paper demonstrates the library’s utility using standard computer vision benchmarks:

  • MNIST: Handwritten digits.
  • FashionMNIST: Clothing articles.
  • CIFAR-10: General objects (airplanes, cars, birds, etc.).

These datasets are used to generate "tempering curves," showing how different priors affect performance across various temperatures.

Key Insights and Ideas

  • The Prior is the Culprit: The library helped confirm that the "cold posterior effect" is largely driven by poor prior choice. When you move away from simple Gaussians, the need to "cool" the posterior often diminishes or disappears.
  • Architecture Matters for Priors: The authors found that heavy-tailed priors (like Student-t or Cauchy) tend to work best for fully-connected networks, while correlated priors are superior for Convolutional Neural Networks (CNNs).
  • Modularity is Key: By using a modular taxonomy, BNNpriors allows researchers to create hierarchical models where the hyperparameters of one prior are themselves governed by another prior.
  • Scalable Accuracy: The library proves that you don't have to sacrifice Bayesian rigor for scale; using cyclical learning rates and preconditioning allows for high-quality posterior samples even in deep architectures.

Key Results

The authors provide quantitative evidence (visualized in the paper's tempering curves) that choosing the right prior leads to:

  1. Lower Error Rates: Better predictive accuracy compared to standard Gaussian BNNs.
  2. Improved Likelihood: Better fit to the data distribution.
  3. Better Calibration: The model's confidence more accurately reflects its actual probability of being correct (measured via Expected Calibration Error).
  4. Robust OOD Detection: Improved ability to identify "Out-of-Distribution" data that the model hasn't seen before.

Specifically, the results show that non-Gaussian priors can significantly alleviate the performance gap typically filled by artificial posterior cooling.

Limitations and Future Work

While powerful, the authors acknowledge several areas for growth:

  • Inference Constraints: Currently, hierarchical priors only support joint inference of weights and hyperparameters. The authors hope to add Gibbs sampling or reversible jump MCMC in the future.
  • Architectural Scope: The library has been primarily tested on standard feed-forward and convolutional networks. Its performance on Recurrent Neural Networks (RNNs) or Transformers remains an open question.
  • Model Selection: Estimating "marginal likelihood" (the evidence for a specific model/prior) is still a difficult challenge that the authors aim to address to help automate the choice of the best prior.

The project is open-source and welcomes community contributions to expand the library's capabilities.

Summary updated

Abstract

Bayesian neural networks have shown great promise in many applications where calibrated uncertainty estimates are crucial and can often also lead to a higher predictive performance. However, it remains challenging to choose a good prior distribution over their weights. While isotropic Gaussian priors are often chosen in practice due to their simplicity, they do not reflect our true prior beliefs well and can lead to suboptimal performance. Our new library, BNNpriors, enables state-of-the-art Markov Chain Monte Carlo inference on Bayesian neural networks with a wide range of predefined priors, including heavy-tailed ones, hierarchical ones, and mixture priors. Moreover, it follows a modular approach that eases the design and implementation of new custom priors. It has facilitated foundational discoveries on the nature of the cold posterior effect in Bayesian neural networks and will hopefully catalyze future research as well as practical applications in this area.
BibTeX
@article{fortuin2021bnnpriors,
  author = {Vincent Fortuin and Adrià Garriga-Alonso and Mark van der Wilk and Laurence Aitchison},
  title = {{BNNpriors: A library for Bayesian neural network inference with different prior distributions}},
  journal = {Softw. Impacts},
  year = {2021},
  doi = {10.1016/j.simpa.2021.100079},
}

Loading related content...

Figure