Efficient Low Rank Gaussian Variational Inference for Neural Networks

Marcin Tomczak, Siddharth Swaroop, Richard E. TurnerNeurIPS 2020 · OpenReview

Introduction

Bayesian Neural Networks (BNNs) are powerful tools for estimating epistemic uncertainty—knowing when the model doesn't know. However, they need efficient ways to estimate this uncertainty without ignoring the correlations between network weights. This paper introduces a scalable Variational Inference (VI) algorithm that uses a low-rank plus diagonal covariance structure, extending the Local Reparametrization Trick to make correlated weight sampling computationally feasible for deep networks like ResNets.

The Problem with Mean-Field VI

To train BNNs, researchers typically use Variational Inference (VI) to approximate the complex posterior distribution of the network's weights. The most common approach, Mean-Field VI (MF-VI), assumes that every single weight in the network is completely independent of the others. This is a massive oversimplification that often leads to pathologies like severe underfitting.

The obvious fix is to model the correlations between weights using a full Gaussian covariance matrix. Unfortunately, doing so causes computational and memory costs to explode. Furthermore, naive methods of sampling these correlated weights introduce massive variance into the gradient estimates, making training unstable. Previous attempts to solve this either relied on heavy approximations or couldn't scale beyond tiny, shallow networks.

The ELRG-VI Solution

The authors introduce Efficient Low Rank Gaussian VI (ELRG-VI), which makes correlated posteriors scalable through three main technical contributions:

  1. Low-Rank + Diagonal Covariance: Instead of a full covariance matrix, the posterior is restricted to a low-rank matrix plus a diagonal matrix. This captures the most important weight correlations while keeping the parameter count manageable.
  2. Extended Local Reparametrization Trick (LRT): The original LRT saves memory and reduces gradient variance by sampling the activations of a layer rather than the weights themselves. However, LRT was previously only mathematically viable for independent (Mean-Field) weights. The authors successfully derive an extension of LRT that works for their low-rank correlated structure.
  3. Efficient KL Divergence: Calculating the complexity penalty (KL divergence) between the approximate posterior and the prior usually requires computing an expensive matrix determinant. The authors derive an analytical, computationally cheap formula that avoids this, reducing the time complexity from $O(N^6)$ to $O(K^3)$, where $K$ is the rank.

They also adapt this approach for Convolutional Neural Networks (CNNs) by treating the low-rank component as reparameterizable while keeping the diagonal variance as a small, fixed constant.

Key Insights and Training Dynamics

  • Sample sharing ruins performance: The naive way to do VI is to sample one set of weights and use it for an entire batch of data. The authors prove empirically that this causes high gradient variance and terrible predictive performance. Using the LRT to generate unique samples per input is strictly necessary to stabilize training.
Comparison of ELRG-VI vs. shared samples showing massive variance when samples are shared.
  • Constant diagonals prevent underfitting: The authors tested two variants: one where the diagonal variance is learned (ELRG-D-VI) and one where it is fixed to a small constant (ELRG-VI). Counterintuitively, learning the diagonal caused larger networks to underfit. Fixing the diagonal acts as a regularizer, allowing the low-rank terms to shine and drastically improving performance on deep networks.
Training curves comparing ELRG-D-VI, ELRG-VI, and MF-VI. ELRG-VI shows stable convergence compared to the pathologies of MF-VI.
  • Better out-of-distribution awareness: Standard Maximum A Posteriori (MAP) training often achieves high accuracy but is wildly overconfident on unfamiliar data. ELRG-VI strikes a much better balance, yielding appropriately high entropy (uncertainty) on out-of-distribution data (like testing a CIFAR100 model on SVHN images) while maintaining accuracy on known data.

Key Results

  • Computational Speedup: The memory and time complexity to update a layer's parameters was reduced from scaling with the cube of the layer dimensions ($O(N_{in}^3 N_{out}^3)$) to scaling linearly with the layer dimensions and the cube of the rank ($O(K^3)$).
  • Vectorized MNIST (MLP): ELRG-VI (Rank $K=5$) achieved a 1.54% error rate, cleanly beating Mean-Field VI (2.51%) and other baselines like K-TIED.
  • CIFAR10 (ResNet18): ELRG-VI scaled successfully to ResNets, achieving 87.24% accuracy. This vastly outperformed MF-VI (76.63%) and beat competing correlated VI methods like VOGN (84.27%).
  • Calibration: On CIFAR10 with ResNet18, ELRG-VI achieved an Expected Calibration Error (ECE) of 0.018, significantly better than MAP (0.080) and MF-VI (0.071), proving its uncertainty estimates are highly reliable.

Limitations and Future Work

  • Choosing the Rank ($K$): The theoretical quality of the approximation depends heavily on the true (and unknown) posterior covariance. Because of this, there is no easy way to select the optimal rank $K$ other than empirical trial and error.
  • CNN Approximations: Applying the LRT to convolutional layers is described as a "crude approximation." It ignores the correlations induced by sliding the exact same kernel over different spatial patches of an image.
  • Runtime Overhead: While vastly more scalable than full-covariance VI, the algorithm's runtime still scales linearly with the chosen rank $K$. Using very high-rank approximations on massive modern architectures remains impractical.

Summary updated

Abstract

Bayesian neural networks are enjoying a renaissance driven in part by recent advances in variational inference (VI). The most common form of VI employs a fully factorized or mean-field distribution, but this is known to suffer from several pathologies, especially as we expect posterior distributions with highly correlated parameters. Current algorithms that capture these correlations with a Gaussian approximating family are difficult to scale to large models due to computational costs and high variance of gradient updates. By using a new form of the reparametrization trick, we derive a computationally efficient algorithm for performing VI with a Gaussian family with a low-rank plus diagonal covariance structure. We scale to deep feed-forward and convolutional architectures. We find that adding low-rank terms to parametrized diagonal covariance does not improve predictive performance except on small networks, but low-rank terms added to a constant diagonal covariance improves performance on small and large-scale network architectures.
BibTeX
@inproceedings{tomczak2020efficient,
  author = {Marcin Tomczak and Siddharth Swaroop and Richard E. Turner},
  title = {{Efficient Low Rank Gaussian Variational Inference for Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=PJfA-zpqUl},
}
Figure