Radial Bayesian Neural Networks: Robust Variational Inference In Big Models

Sebastian Farquhar, Michael Osborne, Yarin Gal

Abstract

We propose Radial Bayesian Neural Networks (BNNs): a variational approximate posterior for BNNs which scales well to large models while maintaining a distribution over weight-space with full support. Other scalable Bayesian deep learning methods, like MC dropout or deep ensembles, have discrete support-they assign zero probability to almost all of the weight-space. Unlike these discrete support methods, Radial BNNs' full support makes them suitable for use as a prior for sequential inference. In addition, they solve the conceptual challenges with the a priori implausibility of weight distributions with discrete support. The Radial BNN is motivated by avoiding a sampling problem in 'mean-field' variational inference (MFVI) caused by the so-called 'soap-bubble' pathology of multivariate Gaussians. We show that, unlike MFVI, Radial BNNs are robust to hyperparameters and can be efficiently applied to a challenging real-world medical application without needing ad-hoc tweaks and intensive tuning. In fact, in this setting Radial BNNs out-perform discrete-support methods like MC dropout. Lastly, by using Radial BNNs as a theoretically principled, robust alternative to MFVI we make significant strides in a Bayesian continual learning evaluation.

Introduction

Bayesian Neural Networks (BNNs) offer a principled framework for handling uncertainty, yet scaling them to modern deep architectures remains a significant challenge. The standard approach, Mean-Field Variational Inference (MFVI), typically employs multivariate Gaussian distributions to approximate the posterior. However, in high-dimensional parameter spaces, Gaussians suffer from the "soap-bubble" pathology: probability mass concentrates in a thin shell far from the mean. This leads to unrepresentative samples during training, resulting in exploding gradient variance and unstable optimization.

Furthermore, popular alternatives like MC dropout and deep ensembles possess "discrete support," assigning zero probability to most of the weight space. This overconfidence makes them unsuitable as priors for continual learning, where a zero-probability prior cannot be updated by new data.

Methods: The Radial BNN

To address these issues, we propose the Radial BNN, which models the approximate posterior in hyperspherical coordinates (radius and angles) rather than standard Cartesian coordinates.

  • Radial Component: We choose a distribution for the radius ($r$) that places mass near the mean, specifically $r = |\tilde{r}|$ where $\tilde{r} \sim \mathcal{N}(0, 1)$. This directly eliminates the soap-bubble effect seen in standard MFVI.
  • Angular Component: We assume a uniform distribution over the hypersphere, ensuring all directions are equally likely.
  • Efficiency: Sampling is computationally efficient; by normalizing a standard Gaussian noise vector and scaling by the radial term, we can sample in weight space without expensive transformations.
  • Objective Function: The entropy term of the Evidence Lower Bound (ELBO) for Radial BNNs is identical to the Gaussian case up to a constant, allowing for easy integration into existing training pipelines.
Figure 1: Comparison of probability mass concentration between MFVI and Radial BNNs.

Empirical Evaluation

We evaluated Radial BNNs across three distinct scales: a large-scale medical imaging task (Diabetic Retinopathy), a sequential learning benchmark (Split-FashionMNIST), and standard small-scale UCI regression datasets.

Robustness and Stability

A key hurdle for MFVI is its extreme sensitivity to initialization. Radial BNNs demonstrate significantly higher robustness; 44% of random hyperparameter configurations reached good performance, compared to only 11% for MFVI. This stability is rooted in the management of gradient variance. As shown in Figure 6, MFVI experiences a dramatic spike in gradient variance as the initialization of $\sigma$ increases, whereas Radial BNNs remain stable.

Figure 6: Gradient variance vs. initialization of sigma. MFVI explodes while Radial BNN remains stable.

Key Results

  • Performance: On the diabetic retinopathy task, a single Radial BNN (AUC 94.3%) outperformed both MC dropout (93.0%) and standard MFVI. An ensemble of Radial BNNs achieved 94.5%, surpassing standard Deep Ensembles (93.9%).
  • Calibration: Radial BNNs are highly calibrated, with predicted probabilities closely matching actual frequencies, a significant advantage over often-overconfident dropout and ensemble methods.
  • Continual Learning: In the Split-FashionMNIST task, Radial BNNs preserved accuracy on early tasks much better than the MFVI-based Variational Continual Learning (VCL) baseline, demonstrating the benefit of "full support" priors.

Limitations

While Radial BNNs excel in high-dimensional settings, they perform similarly to MFVI on very small datasets (UCI) and do not outperform specialized low-dimensional methods. Additionally, while they improve continual learning, the "single-headed" setting remains a challenge. Future work will address a missing term in the initial derivation of the cross-entropy term for a Radial prior.

Summary updated

Abstract

We propose Radial Bayesian Neural Networks (BNNs): a variational approximate posterior for BNNs which scales well to large models while maintaining a distribution over weight-space with full support. Other scalable Bayesian deep learning methods, like MC dropout or deep ensembles, have discrete support-they assign zero probability to almost all of the weight-space. Unlike these discrete support methods, Radial BNNs' full support makes them suitable for use as a prior for sequential inference. In addition, they solve the conceptual challenges with the a priori implausibility of weight distributions with discrete support. The Radial BNN is motivated by avoiding a sampling problem in 'mean-field' variational inference (MFVI) caused by the so-called 'soap-bubble' pathology of multivariate Gaussians. We show that, unlike MFVI, Radial BNNs are robust to hyperparameters and can be efficiently applied to a challenging real-world medical application without needing ad-hoc tweaks and intensive tuning. In fact, in this setting Radial BNNs out-perform discrete-support methods like MC dropout. Lastly, by using Radial BNNs as a theoretically principled, robust alternative to MFVI we make significant strides in a Bayesian continual learning evaluation.
BibTeX
@inproceedings{farquhar2020radial,
  author = {Sebastian Farquhar and Michael Osborne and Yarin Gal},
  title = {{Radial Bayesian Neural Networks: Robust Variational Inference In Big Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=KqreW1fKTpg},
}
Figure