Collapsed Variational Bounds for Bayesian Neural Networks

Marcin Tomczak, Siddharth Swaroop, Andrew Y. K. Foong, Richard E. TurnerNeurIPS 2021 · OpenReview

Introduction

Bayesian Neural Networks (BNNs) offer a principled approach to handling uncertainty in deep learning. However, training them using Variational Inference—specifically the popular Mean-Field Variational Inference (MF-VI)—is notoriously difficult. Standard BNNs often suffer from severe underfitting because they are overly sensitive to the choice of weight priors.

This paper introduces "collapsed" variational bounds that analytically marginalize out prior parameters. This creates a tighter training objective that prevents the model from prematurely "pruning" useful weights, significantly boosting predictive performance in deep architectures.

The Problem: Over-Pruning in MF-VI

In deep networks, MF-VI tends to be "under-confident" and prone to underfitting. This occurs because the standard training objective—the Evidence Lower Bound (ELBO)—is highly sensitive to the weight prior, which is typically a fixed Gaussian centered at zero.

If this prior is too "strong," the optimization process aggressively pushes weight distributions toward zero, a phenomenon known as over-pruning. These pruned weights then act as random noise, drowning out useful signals and degrading the model's accuracy. To circumvent this, researchers usually resort to manual tuning or heuristics to find a workable prior, which is inefficient and often ineffective for large-scale models.

The Solution: Collapsed Variational Bounds

To address this, the authors propose treating the prior parameters (the means and variances of the weight distributions) as latent variables rather than fixed constants.

Instead of learning these prior parameters via standard gradient descent alongside the weights, the authors use the variational principle to analytically solve for their optimal distribution. By substituting these optimal solutions back into the ELBO, they derive a "collapsed" bound ($\mathcal{L}^*$) that is mathematically tighter than the standard version.

Comparison of how different bounds regularize the posterior. The standard bound (left) strongly prunes weights toward the prior, while collapsed bounds (middle, right) offer more flexibility.

They introduce two main variants:

  1. CM-MF-VI ($\mathcal{L}^*_m$): Collapses the bound with respect to prior means while keeping variances fixed.
  2. CMV-MF-VI ($\mathcal{L}^*_{mv}$): Collapses the bound with respect to both prior means and variances.

Crucially, these new objectives do not add any significant computational or memory overhead compared to standard MF-VI, making them easy "drop-in" replacements for existing BNN training pipelines.

Key Insights and Results

The authors evaluated their methods across a wide range of benchmarks, including UCI Regression datasets, image classification tasks (MNIST, CIFAR-10, etc.), and modern architectures like ResNet-18.

  • Fixing Over-Pruning: The collapsed bound effectively "weakens" the regularization on the posterior means. This allows the model to shift the biases of pruned units to values that saturate activations (like ReLU), essentially "turning off" the noise from pruned weights rather than letting it leak into the final prediction.
  • Tighter is Better for Tuning: The collapsed ELBO serves as a much more reliable signal for tuning hyperparameters (like the strength of the prior) than the standard ELBO. The peak of the collapsed ELBO aligns closely with the best test performance.
  • MF-VI is More Capable Than We Thought: Contrary to the growing sentiment that Mean-Field VI cannot work for deep models, this paper demonstrates that treating priors correctly allows MF-VI to outperform many complex baselines.
Test log-likelihood across various UCI regression datasets. The collapsed bounds (red) consistently outperform standard MF-VI (blue) as network depth increases.

Empirically, the results show a dramatic improvement over standard MF-VI:

  • MNIST: CM-MF-VI achieved a test Negative Log-Likelihood (NLL) of 0.047, compared to 0.094 for standard MF-VI.
  • CIFAR-10 (ResNet-18): The collapsed bound reduced the error rate to 13.34%, a massive leap over the 22.92% error seen with standard MF-VI.
  • Out-of-Distribution (OOD) Robustness: Despite fitting the training data much better, the models retained their "Bayesian-ness." When evaluated on out-of-distribution data, the collapsed models correctly exhibited high predictive uncertainty.

Limitations and Future Work

While the method eliminates the need to tune prior parameters directly, it introduces "hyper-priors" (such as $\alpha_{reg}$). Fortunately, the authors demonstrate that these are much easier to tune because the ELBO itself acts as a reliable guide.

Additionally, the paper still operates within the mean-field framework, assuming weights are independent. Exploring how collapsed bounds interact with more expressive posterior distributions (like matrix-variate or low-rank Gaussians) is a promising next step. Finally, while scaling to ResNet-18 is a significant milestone, testing these bounds on massive transformer-based models remains an open challenge for the community.

Summary updated

Abstract

Recent interest in learning large variational Bayesian Neural Networks (BNNs) has been partly hampered by poor predictive performance caused by underfitting, and their performance is known to be very sensitive to the prior over weights. Current practice often fixes the prior parameters to standard values or tunes them using heuristics or cross-validation. In this paper, we treat prior parameters in a distributional way by extending the model and collapsing the variational bound with respect to their posteriors. This leads to novel and tighter Evidence Lower Bounds (ELBOs) for performing variational inference (VI) in BNNs. Our experiments show that the new bounds significantly improve the performance of Gaussian mean-field VI applied to BNNs on a variety of data sets, demonstrating that mean-field VI works well even in deep models. We also find that the tighter ELBOs can be good optimization targets for learning the hyperparameters of hierarchical priors.
BibTeX
@inproceedings{tomczak2021collapsed,
  author = {Marcin Tomczak and Siddharth Swaroop and Andrew Y. K. Foong and Richard E. Turner},
  title = {{Collapsed Variational Bounds for Bayesian Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=LqwD_ia9DuG},
}
Figure