Learning Structural Weight Uncertainty for Sequential Decision-Making

Ruiyi Zhang, Chunyuan Li, Changyou Chen, Lawrence Carin

Abstract

Learning probability distributions on the weights of neural networks (NNs) has recently proven beneficial in many applications. Bayesian methods, such as Stein variational gradient descent (SVGD), provide a framework for this, but often rely on simplifying assumptions like independent priors and factorized posteriors. This paper introduces Structural Stein Variational Gradient Descent ($S^2VGD$), a new framework for Bayesian Neural Networks (BNNs) that captures complex intra-layer dependencies through Matrix Variate Gaussian (MVG) priors and efficient reparameterization using Householder flows. We demonstrate that learning this "structural uncertainty" significantly improves performance in sequential decision-making tasks, such as contextual bandits and reinforcement learning, by enabling more effective exploration and better management of the exploration-exploitation tradeoff.

Introduction

Deep learning models are notoriously overconfident. Standard neural networks typically employ point estimation to find a single "best" set of weights, which fails to quantify how uncertain the model is about its own predictions. Bayesian Neural Networks (BNNs) attempt to mitigate this by learning a probability distribution over weights. However, traditional BNNs often make two restrictive assumptions for mathematical convenience:

  1. Independent Priors: Weights are assumed to be independent (e.g., isotropic Gaussians), ignoring natural correlations between neurons in the same layer.
  2. Factorized Posteriors: The resulting distributions are assumed to be simple and uncorrelated, which leads to a significant underestimation of model uncertainty.

In sequential decision-making environments, such as robotics or recommendation systems, a model that lacks awareness of its own uncertainty will fail to explore effectively, leading to sub-optimal long-term performance.

Methods: Structural Stein Variational Gradient Descent

The proposed $S^2VGD$ framework addresses these limitations by combining Matrix Variate Gaussian (MVG) priors with a novel reparameterization technique to capture "structural uncertainty."

Matrix Variate Gaussian Priors

Instead of treating weights as a flat list of independent variables, $S^2VGD$ treats them as a matrix where rows and columns have their own covariance. This allows the model to learn intra-layer correlations that are typically ignored in standard BNNs.

Reparameterization with Householder Flows

Learning full covariance matrices is computationally expensive ($O(N^2)$). To maintain efficiency, the authors decompose the weight matrix into five components: a standard Gaussian matrix, two diagonal scaling matrices, and two orthogonal rotation matrices.

Structural decomposition of weight matrices (a) and efficient implementation via Householder flows (b).

By using Householder transformations to represent these rotations, the model captures structural information without the parameter explosion associated with full covariance matrices. This structural prior is integrated into Stein Variational Gradient Descent (SVGD), a non-parametric method that uses a set of "particles" to approximate the posterior distribution without assuming a fixed functional form.

Experimental Results

The model was evaluated across diverse tasks, including regression (UCI datasets), classification (MNIST), contextual bandits (Yahoo! Today Module), and reinforcement learning (OpenAI rllab).

Performance in Sequential Decision-Making

In contextual bandit tasks using the Yahoo! news recommendation dataset, $S^2VGD$ demonstrated superior performance. As shown in the figure below, $S^2VGD$ achieved a significantly higher Click-Through Rate (CTR) compared to the classic LinUCB algorithm and standard SVGD, particularly as the number of visits increased.

Click-Through Rate (CTR) performance on the Yahoo! Today Module dataset, showing S2VGD outperforming standard SVGD and LinUCB over 40 million visits.

Key Insights

  • Structure vs. Capacity: The authors proved that the performance gain was due to the representation of uncertainty rather than simply increasing model parameters. A standard BNN with more hidden units still performed worse than a smaller $S^2VGD$ model.
  • Efficient Exploration: By capturing internal weight correlations, $S^2VGD$ maintains a more diverse set of hypotheses about the environment. This leads to lower "regret" in bandit tasks and faster convergence to higher rewards in reinforcement learning tasks like the Double Pendulum.

Conclusion

$S^2VGD$ provides a robust framework for learning structural weight uncertainty in BNNs. By moving beyond the assumption of independent weights, it enables more intelligent exploration in complex, sequential environments. While the method introduces some computational overhead and hyperparameter sensitivity regarding the number of particles ($M$) and transformations ($K$), its superior performance in real-world recommendation and control tasks highlights the critical importance of structural priors in Bayesian deep learning.

Summary updated

Abstract

Learning probability distributions on the weights of neural networks (NNs) has recently proven beneficial in many applications. Bayesian methods, such as Stein variational gradient descent (SVGD), ...
BibTeX
@inproceedings{zhang2018learning,
  author = {Ruiyi Zhang and Chunyuan Li and Changyou Chen and Lawrence Carin},
  title = {{Learning Structural Weight Uncertainty for Sequential Decision-Making}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2018},
  url = {https://openreview.net/forum?id=ETBhKpJNlb},
}
Figure