Introduction
Bayesian Neural Networks (BNNs) are powerful tools for estimating epistemic uncertainty—knowing when the model doesn't know. However, they need efficient ways to estimate this uncertainty without ignoring the correlations between network weights. This paper introduces a scalable Variational Inference (VI) algorithm that uses a low-rank plus diagonal covariance structure, extending the Local Reparametrization Trick to make correlated weight sampling computationally feasible for deep networks like ResNets.
The Problem with Mean-Field VI
To train BNNs, researchers typically use Variational Inference (VI) to approximate the complex posterior distribution of the network's weights. The most common approach, Mean-Field VI (MF-VI), assumes that every single weight in the network is completely independent of the others. This is a massive oversimplification that often leads to pathologies like severe underfitting.
The obvious fix is to model the correlations between weights using a full Gaussian covariance matrix. Unfortunately, doing so causes computational and memory costs to explode. Furthermore, naive methods of sampling these correlated weights introduce massive variance into the gradient estimates, making training unstable. Previous attempts to solve this either relied on heavy approximations or couldn't scale beyond tiny, shallow networks.
The ELRG-VI Solution
The authors introduce Efficient Low Rank Gaussian VI (ELRG-VI), which makes correlated posteriors scalable through three main technical contributions:
- Low-Rank + Diagonal Covariance: Instead of a full covariance matrix, the posterior is restricted to a low-rank matrix plus a diagonal matrix. This captures the most important weight correlations while keeping the parameter count manageable.
- Extended Local Reparametrization Trick (LRT): The original LRT saves memory and reduces gradient variance by sampling the activations of a layer rather than the weights themselves. However, LRT was previously only mathematically viable for independent (Mean-Field) weights. The authors successfully derive an extension of LRT that works for their low-rank correlated structure.
- Efficient KL Divergence: Calculating the complexity penalty (KL divergence) between the approximate posterior and the prior usually requires computing an expensive matrix determinant. The authors derive an analytical, computationally cheap formula that avoids this, reducing the time complexity from $O(N^6)$ to $O(K^3)$, where $K$ is the rank.
They also adapt this approach for Convolutional Neural Networks (CNNs) by treating the low-rank component as reparameterizable while keeping the diagonal variance as a small, fixed constant.
Key Insights and Training Dynamics
- Sample sharing ruins performance: The naive way to do VI is to sample one set of weights and use it for an entire batch of data. The authors prove empirically that this causes high gradient variance and terrible predictive performance. Using the LRT to generate unique samples per input is strictly necessary to stabilize training.
- Constant diagonals prevent underfitting: The authors tested two variants: one where the diagonal variance is learned (
ELRG-D-VI) and one where it is fixed to a small constant (ELRG-VI). Counterintuitively, learning the diagonal caused larger networks to underfit. Fixing the diagonal acts as a regularizer, allowing the low-rank terms to shine and drastically improving performance on deep networks.
- Better out-of-distribution awareness: Standard Maximum A Posteriori (MAP) training often achieves high accuracy but is wildly overconfident on unfamiliar data. ELRG-VI strikes a much better balance, yielding appropriately high entropy (uncertainty) on out-of-distribution data (like testing a CIFAR100 model on SVHN images) while maintaining accuracy on known data.
Key Results
- Computational Speedup: The memory and time complexity to update a layer's parameters was reduced from scaling with the cube of the layer dimensions ($O(N_{in}^3 N_{out}^3)$) to scaling linearly with the layer dimensions and the cube of the rank ($O(K^3)$).
- Vectorized MNIST (MLP): ELRG-VI (Rank $K=5$) achieved a 1.54% error rate, cleanly beating Mean-Field VI (2.51%) and other baselines like K-TIED.
- CIFAR10 (ResNet18): ELRG-VI scaled successfully to ResNets, achieving 87.24% accuracy. This vastly outperformed MF-VI (76.63%) and beat competing correlated VI methods like VOGN (84.27%).
- Calibration: On CIFAR10 with ResNet18, ELRG-VI achieved an Expected Calibration Error (ECE) of 0.018, significantly better than MAP (0.080) and MF-VI (0.071), proving its uncertainty estimates are highly reliable.
Limitations and Future Work
- Choosing the Rank ($K$): The theoretical quality of the approximation depends heavily on the true (and unknown) posterior covariance. Because of this, there is no easy way to select the optimal rank $K$ other than empirical trial and error.
- CNN Approximations: Applying the LRT to convolutional layers is described as a "crude approximation." It ignores the correlations induced by sliding the exact same kernel over different spatial patches of an image.
- Runtime Overhead: While vastly more scalable than full-covariance VI, the algorithm's runtime still scales linearly with the chosen rank $K$. Using very high-rank approximations on massive modern architectures remains impractical.
Summary updated