Radial and Directional Posteriors for Bayesian Deep Learning

ChangYong Oh, Kamil Adamczewski, Mijung Park

Abstract

We propose a new variational family for Bayesian neural networks. We decompose the variational posterior into two components, where the radial component captures the strength of each neuron in terms of its magnitude; while the directional component captures the statistical dependencies among the weight parameters. The dependencies learned via the directional density provide better modeling performance compared to the widely-used Gaussian mean-field-type variational family. In addition, the strength of input and output neurons learned via our posterior provides a structured way to compress neural networks. Indeed, experiments show that our variational family improves predictive performance and yields compressed networks simultaneously.

Introduction

Deep learning models are often over-parameterized and prone to overconfidence in their predictions. Bayesian Neural Networks (BNNs) address this by treating weights as probability distributions. However, standard BNNs typically rely on "mean-field" Gaussian approximations, which assume weight independence—a significant oversimplification given the high correlation of weights in neural networks. Furthermore, uncertainty quantification and model compression are often treated as separate challenges. This paper introduces the Radial and Directional Posterior (RDP) to address both by rethinking weight space geometry.

The Radial and Directional Posterior (RDP)

The core innovation of this work is the decomposition of weight vectors into magnitude (radial) and orientation (directional) components.

Probabilistic graphical model of the RDP framework, illustrating the decomposition of weights into radial (magnitude) and directional (orientation) components.
  1. Directional Component: The model employs the von Mises-Fisher (vMF) distribution, effectively a "Gaussian on a hypersphere," to capture statistical dependencies between weights within a row or column.
  2. Radial Component: A log-Normal posterior is paired with a Half-Cauchy prior, functioning as a "Group Horseshoe" prior that naturally encourages sparsity by pushing unnecessary neuron magnitudes toward zero.

Overcoming Numerical Instability

Implementing vMF distributions in high-dimensional spaces is challenging due to the numerical instability of modified Bessel function ratios. The authors contribute a novel, stable approximation for these ratios, enabling the model to scale to modern deep learning architectures with thousands of neurons.

Uncertainty Quantification and Model Compression

By separating a neuron's strength (radius) from its feature sensitivity (direction), the RDP framework offers a more intuitive approach to uncertainty. In regions with sparse data, RDP exhibits significantly higher uncertainty than standard methods like Bayes By Backprop (BBB), enhancing reliability in real-world decision-making.

Comparison of uncertainty quantification on a 1D synthetic task between Bayes By Backprop (BBB), a Radius-only model, and the proposed RDP.

Simultaneously, RDP facilitates structured pruning. Unlike unstructured methods that remove individual weights, the radial component applies to entire rows or columns, allowing the model to deactivate entire neurons. This leads to substantial reductions in FLOPs and improved hardware efficiency.

Experimental Evaluation

The authors evaluated RDP across regression, uncertainty visualization, and image classification tasks:

  • Regression: RDP outperformed baselines like Dropout and Variational Matrix Gaussian in test log-likelihood on 6 out of 10 UCI datasets.
  • Uncertainty & Exploration: In the "Wheel" contextual bandit task, RDP achieved a cumulative regret of 2.4, significantly lower than the 3.6 achieved by a radial-only model, demonstrating the importance of capturing weight directions for effective exploration.
  • Compression: On MNIST, RDP compressed LeNet-5 to 20K parameters with only 1.0% error. On VGG16 (CIFAR-10), it pruned convolutional layers more aggressively than competitors, notably reducing the computational footprint.

Conclusion

The Radial and Directional Posterior family provides a robust framework for simultaneous uncertainty quantification and structured model compression. While the vMF reparameterization trick involves complex rejection sampling, the benefits in predictive performance and efficiency are significant. Future work includes extending this approach to attention mechanisms in Transformers and exploring alternative directional distributions.

Summary updated

Abstract

We propose a new variational family for Bayesian neural networks. We decompose the variational posterior into two components, where the radial component captures the strength of each neuron in terms of its magnitude; while the directional component captures the statistical dependencies among the weight parameters. The dependencies learned via the directional density provide better modeling performance compared to the widely-used Gaussian mean-field-type variational family. In addition, the strength of input and output neurons learned via our posterior provides a structured way to compress neural networks. Indeed, experiments show that our variational family improves predictive performance and yields compressed networks simultaneously.
BibTeX
@inproceedings{oh2020radial,
  author = {ChangYong Oh and Kamil Adamczewski and Mijung Park},
  title = {{Radial and Directional Posteriors for Bayesian Deep Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=V1db7OAUsM},
}
Figure