Abstract
Approximate inference in Bayesian deep networks exhibits a dilemma of how to yield high fidelity posterior approximations while maintaining computational efficiency and scalability. We tackle this challenge by introducing a novel variational structured approximation inspired by the Bayesian interpretation of Dropout regularization. Concretely, we focus on the inflexibility of the factorized structure in Dropout posterior and then propose an improved method called Variational Structured Dropout (VSD). VSD employs an orthogonal transformation to learn a structured representation on the variational Gaussian noise with plausible complexity, and consequently induces statistical dependencies in the approximate posterior. Theoretically, VSD successfully addresses the pathologies of previous Variational Dropout methods and thus offers a standard Bayesian justification. We further show that VSD induces an adaptive regularization term with several desirable properties which contribute to better generalization. Finally, we conduct extensive experiments on standard benchmarks to demonstrate the effectiveness of VSD over state-of-the-art variational methods on predictive accuracy, uncertainty estimation, and out-of-distribution detection.
Introduction
Deep learning models are notoriously overconfident, often making incorrect predictions with high certainty. Bayesian Neural Networks (BNNs) offer a principled solution by treating weights as distributions, enabling better uncertainty quantification. However, BNNs face a "dilemma of fidelity vs. efficiency." Standard Mean-Field Variational Inference (MFVI) is computationally efficient but ignores weight correlations, leading to poor uncertainty estimates. Conversely, structured approximations that capture these dependencies often fail to scale to modern architectures.
Variational Dropout (VD) emerged as a popular middle ground, yet it suffers from "theoretical pathologies." Previous VD methods often utilize improper priors or result in "singular" distributions where the Kullback-Leibler (KL) divergence becomes undefined. This paper introduces Variational Structured Dropout (VSD), a framework that maintains the efficiency of Dropout while providing a rigorous Bayesian foundation and improved performance.
Methodology: Variational Structured Dropout
The core innovation of VSD lies in learning structured dependencies within the variational posterior without sacrificing scalability.
Householder Transformations
Instead of using simple diagonal Gaussian noise, VSD applies a series of Householder reflections. This orthogonal transformation allows the model to learn a structured covariance matrix for the noise with very little computational overhead. By "rotating" the noise, VSD captures statistical dependencies between neurons, effectively bridging the gap between mean-field and fully structured approximations.
Hierarchical Prior
VSD incorporates a two-level hierarchical prior where a latent variable $z$ serves as a global scaling factor for the weights. This hierarchy increases the model's expressiveness and allows it to adaptively tune the level of stochasticity across different layers.
Addressing Theoretical Pathologies
Unlike prior VD implementations, VSD employs a proper prior and ensures the approximate posterior is non-singular. This satisfies the "KL condition," providing a solid Bayesian justification for the method and ensuring the Evidence Lower Bound (ELBO) objective is well-defined.
Theoretical Insights
VSD induces an adaptive regularization term that encourages the model to find "flat minima" in the loss landscape. Flat minima are strongly correlated with superior generalization on unseen data. Furthermore, VSD naturally penalizes the spectral norm and stable rank of weight matrices, which are key theoretical indicators of a model's ability to generalize well.
Experimental Results
The authors evaluated VSD across various benchmarks, including MNIST, CIFAR-10/100, SVHN, and several UCI regression datasets using architectures like ResNet-18.
Predictive Performance and Calibration
VSD consistently outperforms standard VD and other variational baselines. On ResNet-18 trained on SVHN, VSD achieved a lower error rate (3.59%) compared to standard VD (3.89%). Crucially, VSD significantly improves uncertainty calibration, achieving the lowest Expected Calibration Error (ECE) in most settings. This means the model's confidence scores are more representative of its actual accuracy.
Out-of-Distribution (OOD) Detection
VSD demonstrates robust OOD detection. When presented with data from a different distribution (e.g., testing a model trained on SVHN with CIFAR-10 images), VSD exhibits much higher entropy in its predictions compared to standard models. This indicates that the model "knows when it doesn't know," a critical property for safety-critical applications.
Conclusion
Variational Structured Dropout (VSD) successfully resolves the fidelity-efficiency trade-off in Bayesian Deep Learning. By leveraging orthogonal transformations and hierarchical priors, VSD provides a theoretically sound and computationally efficient method for improving predictive accuracy and uncertainty estimation. Future work includes scaling VSD to massive transformer-based architectures and exploring its utility in active learning and reinforcement learning.
Summary updated