Abstract
Learning probability distributions on the weights of neural networks (NNs) has recently proven beneficial in many applications. Bayesian methods, such as Stein variational gradient descent (SVGD), provide a framework for this, but often rely on simplifying assumptions like independent priors and factorized posteriors. This paper introduces Structural Stein Variational Gradient Descent ($S^2VGD$), a new framework for Bayesian Neural Networks (BNNs) that captures complex intra-layer dependencies through Matrix Variate Gaussian (MVG) priors and efficient reparameterization using Householder flows. We demonstrate that learning this "structural uncertainty" significantly improves performance in sequential decision-making tasks, such as contextual bandits and reinforcement learning, by enabling more effective exploration and better management of the exploration-exploitation tradeoff.
Introduction
Deep learning models are notoriously overconfident. Standard neural networks typically employ point estimation to find a single "best" set of weights, which fails to quantify how uncertain the model is about its own predictions. Bayesian Neural Networks (BNNs) attempt to mitigate this by learning a probability distribution over weights. However, traditional BNNs often make two restrictive assumptions for mathematical convenience:
- Independent Priors: Weights are assumed to be independent (e.g., isotropic Gaussians), ignoring natural correlations between neurons in the same layer.
- Factorized Posteriors: The resulting distributions are assumed to be simple and uncorrelated, which leads to a significant underestimation of model uncertainty.
In sequential decision-making environments, such as robotics or recommendation systems, a model that lacks awareness of its own uncertainty will fail to explore effectively, leading to sub-optimal long-term performance.
Methods: Structural Stein Variational Gradient Descent
The proposed $S^2VGD$ framework addresses these limitations by combining Matrix Variate Gaussian (MVG) priors with a novel reparameterization technique to capture "structural uncertainty."
Matrix Variate Gaussian Priors
Instead of treating weights as a flat list of independent variables, $S^2VGD$ treats them as a matrix where rows and columns have their own covariance. This allows the model to learn intra-layer correlations that are typically ignored in standard BNNs.
Reparameterization with Householder Flows
Learning full covariance matrices is computationally expensive ($O(N^2)$). To maintain efficiency, the authors decompose the weight matrix into five components: a standard Gaussian matrix, two diagonal scaling matrices, and two orthogonal rotation matrices.
By using Householder transformations to represent these rotations, the model captures structural information without the parameter explosion associated with full covariance matrices. This structural prior is integrated into Stein Variational Gradient Descent (SVGD), a non-parametric method that uses a set of "particles" to approximate the posterior distribution without assuming a fixed functional form.
Experimental Results
The model was evaluated across diverse tasks, including regression (UCI datasets), classification (MNIST), contextual bandits (Yahoo! Today Module), and reinforcement learning (OpenAI rllab).
Performance in Sequential Decision-Making
In contextual bandit tasks using the Yahoo! news recommendation dataset, $S^2VGD$ demonstrated superior performance. As shown in the figure below, $S^2VGD$ achieved a significantly higher Click-Through Rate (CTR) compared to the classic LinUCB algorithm and standard SVGD, particularly as the number of visits increased.
Key Insights
- Structure vs. Capacity: The authors proved that the performance gain was due to the representation of uncertainty rather than simply increasing model parameters. A standard BNN with more hidden units still performed worse than a smaller $S^2VGD$ model.
- Efficient Exploration: By capturing internal weight correlations, $S^2VGD$ maintains a more diverse set of hypotheses about the environment. This leads to lower "regret" in bandit tasks and faster convergence to higher rewards in reinforcement learning tasks like the Double Pendulum.
Conclusion
$S^2VGD$ provides a robust framework for learning structural weight uncertainty in BNNs. By moving beyond the assumption of independent weights, it enables more intelligent exploration in complex, sequential environments. While the method introduces some computational overhead and hyperparameter sensitivity regarding the number of particles ($M$) and transformations ($K$), its superior performance in real-world recommendation and control tasks highlights the critical importance of structural priors in Bayesian deep learning.
Summary updated