Abstract
We propose a new variational family for Bayesian neural networks. We decompose the variational posterior into two components, where the radial component captures the strength of each neuron in terms of its magnitude; while the directional component captures the statistical dependencies among the weight parameters. The dependencies learned via the directional density provide better modeling performance compared to the widely-used Gaussian mean-field-type variational family. In addition, the strength of input and output neurons learned via the radial density provides a structured way to compress neural networks. Indeed, experiments show that our variational family improves predictive performance and yields compressed networks simultaneously.
Introduction
Standard Bayesian Neural Networks (BNNs) typically rely on the mean-field assumption, which posits that every weight in a network is independent of every other weight. While this simplifies the underlying mathematics, it fails to capture the complex correlations inherent in neural network parameters. In practice, the "meaning" of a specific weight is often inextricably linked to its neighbors within the same neuron or layer. Ignoring these dependencies frequently results in poor uncertainty estimates and sub-optimal predictive performance.
Furthermore, deep learning models are notoriously over-parameterized. While pruning—removing unnecessary weights—is a common technique for model compression, standard BNNs lack a principled, structured mechanism to determine which entire neurons or filters can be safely discarded. This paper introduces the Radial and Directional Posterior (RDP) to address both modeling fidelity and model efficiency.
Methodology
The core innovation of this work is the decomposition of each row of a weight matrix into two distinct parts: a Directional Component and a Radial Component.
- Directional Component: This is modeled using the von Mises-Fisher (vMF) distribution, which defines a probability density on a hypersphere. By sharing a "concentration" parameter across a layer, the model effectively captures how weights within a row correlate with one another.
- Radial Component: This is modeled using a Half-Cauchy prior and a log-Normal posterior. This configuration is inspired by "Horseshoe priors," which are effective at encouraging sparsity in statistical models.
Overcoming Numerical Instability
A significant technical challenge in using vMF distributions for high-dimensional neural networks is the calculation of gradients involving "modified Bessel functions." These functions are notoriously unstable and can lead to training failures. To resolve this, the authors developed a novel algebraic approximation for the ratio of these Bessel functions, ensuring numerical stability and compatibility with modern GPU acceleration.
Structured Sparsity and Compression
Unlike standard methods that prune individual weights (unstructured sparsity), RDP enables structured sparsity. By applying the radial density to entire rows, columns, or both, the model can prune entire neurons or convolutional filters.
This "double-grouping" approach allows the model to learn which specific neurons are redundant and which entire layers can be thinned out. This leads to tangible hardware speedups and significant reductions in memory footprint, rather than just theoretical sparsity.
Experimental Results
The authors evaluated the RDP framework across regression and classification tasks:
- Predictive Performance: On nine standard UCI regression datasets, RDP achieved the highest test log-likelihood on five of them. It consistently outperformed standard Variational Inference (VI) and Variational Matrix Gaussian (VMG) methods, demonstrating the value of modeling weight dependencies.
- Model Compression: On the MNIST dataset using a LeNet-5 architecture, RDP compressed the model down to just 20,000 parameters and 125,000 FLOPs while maintaining a 1.0% error rate. This performance surpasses several state-of-the-art pruning methods, such as Group Lasso and Structured Bayesian Pruning, in balancing model size with accuracy.
Conclusion
The Radial and Directional Posterior offers a robust alternative to mean-field variational families. By explicitly modeling weight orientations and magnitudes, it provides both superior uncertainty quantification and a natural pathway for structured model compression. Future work may explore extending the pruning capabilities to the directional component and scaling the framework to massive architectures like Transformers.
Summary updated