Abstract
We propose Radial Bayesian Neural Networks (BNNs): a variational approximate posterior for BNNs which scales well to large models while maintaining a distribution over weight-space with full support. Other scalable Bayesian deep learning methods, like MC dropout or deep ensembles, have discrete support—they assign zero probability to almost all of the weight-space. Unlike these discrete support methods, Radial BNNs' full support makes them suitable for use as a prior for sequential inference. In addition, they solve the conceptual challenges with the a priori implausibility of weight distributions with discrete support. The Radial BNN is motivated by avoiding a sampling problem in 'mean-field' variational inference (MFVI) caused by the so-called 'soap-bubble' pathology of multivariate Gaussians. We show that, unlike MFVI, Radial BNNs are robust to hyperparameters and can be efficiently applied to a challenging real-world medical application without needing ad-hoc tweaks and intensive tuning. In fact, in this setting Radial BNNs out-perform discrete-support methods like MC dropout. Lastly, by using Radial BNNs as a theoretically principled, robust alternative to MFVI we make significant strides in a Bayesian continual learning evaluation.
The Scaling Dilemma in Bayesian Deep Learning
Bayesian Neural Networks (BNNs) aim to represent uncertainty by learning a distribution over weights rather than a single point estimate. However, researchers face a dilemma when scaling to large models. Popular scalable methods like MC dropout and deep ensembles only assign probability to a few specific points in weight space. This "discrete support" is epistemically pathological—it is implausible that the "true" weights are exactly one of a handful of options. More practically, these methods are poor for continual learning, where a rich, non-zero prior is required to learn new tasks without forgetting old ones.
Alternatively, Mean-Field Variational Inference (MFVI) learns a full distribution (usually a Gaussian) but often fails on large models. The authors identify the culprit: the "soap-bubble" pathology. In high-dimensional spaces, the probability mass of a multivariate Gaussian concentrates in a thin shell far away from the mean.
This phenomenon makes samples unrepresentative of the model's central belief, leading to massive gradient variance that destabilizes the optimization process. Radial BNNs are designed to be as scalable as MFVI while remaining robust enough to work on real-world, high-dimensional problems by "popping" this soap bubble.
Methodology: The Radial Approach
The Radial BNN reparameterizes the approximate posterior in hyperspherical coordinates. Instead of standard Cartesian coordinates, the distribution is defined by:
- The Radius ($r$): The distance from the mean is explicitly modeled. By choosing a distribution where samples are likely to be near the mean, the model avoids the sampling issues of standard Gaussians.
- The Angles: The model assumes a uniform distribution over the hypersphere, meaning all directions are equally likely.
Crucially, this reparameterization allows for efficient sampling with $O(D)$ complexity. One simply takes a standard Gaussian noise vector, normalizes it to unit length to pick a direction, and multiplies it by the sampled radius. This makes Radial BNNs just as computationally efficient as standard MFVI while being significantly more robust to hyperparameters.
Empirical Performance and Calibration
The authors evaluated Radial BNNs on a large-scale Diabetic Retinopathy Prescreening task (~15M parameters). Radial BNNs achieved a ROC-AUC of 94.3%, outperforming both MC dropout (93.0%) and standard MFVI (which failed to converge without significant ad-hoc tweaks).
Beyond accuracy, Radial BNNs demonstrated superior calibration. As shown in the calibration plot, the model's predicted probabilities closely match actual disease frequencies. In contrast, discrete support methods like MC dropout and ensembles tended to be overconfident.
Advancing Continual Learning
Because Radial BNNs maintain full support over weight space, they can serve as a principled "data-dependent prior." In sequential learning tasks (like Split FashionMNIST), the posterior of a completed task becomes the prior for the next. Radial BNNs maintained high accuracy on initial tasks even after learning multiple subsequent categories, whereas MFVI-based baselines suffered from significant catastrophic forgetting.
Conclusion
Radial BNNs offer a theoretically grounded solution to the sampling pathologies that have long hindered variational inference in high dimensions. By providing a scalable method with full weight-space support, they bridge the gap between the computational efficiency of discrete methods and the principled uncertainty of full Bayesian inference.
Summary updated