Introduction
Bayesian Neural Networks (BNNs) offer a principled approach to handling uncertainty in deep learning. However, training them using Variational Inference—specifically the popular Mean-Field Variational Inference (MF-VI)—is notoriously difficult. Standard BNNs often suffer from severe underfitting because they are overly sensitive to the choice of weight priors.
This paper introduces "collapsed" variational bounds that analytically marginalize out prior parameters. This creates a tighter training objective that prevents the model from prematurely "pruning" useful weights, significantly boosting predictive performance in deep architectures.
The Problem: Over-Pruning in MF-VI
In deep networks, MF-VI tends to be "under-confident" and prone to underfitting. This occurs because the standard training objective—the Evidence Lower Bound (ELBO)—is highly sensitive to the weight prior, which is typically a fixed Gaussian centered at zero.
If this prior is too "strong," the optimization process aggressively pushes weight distributions toward zero, a phenomenon known as over-pruning. These pruned weights then act as random noise, drowning out useful signals and degrading the model's accuracy. To circumvent this, researchers usually resort to manual tuning or heuristics to find a workable prior, which is inefficient and often ineffective for large-scale models.
The Solution: Collapsed Variational Bounds
To address this, the authors propose treating the prior parameters (the means and variances of the weight distributions) as latent variables rather than fixed constants.
Instead of learning these prior parameters via standard gradient descent alongside the weights, the authors use the variational principle to analytically solve for their optimal distribution. By substituting these optimal solutions back into the ELBO, they derive a "collapsed" bound ($\mathcal{L}^*$) that is mathematically tighter than the standard version.
They introduce two main variants:
- CM-MF-VI ($\mathcal{L}^*_m$): Collapses the bound with respect to prior means while keeping variances fixed.
- CMV-MF-VI ($\mathcal{L}^*_{mv}$): Collapses the bound with respect to both prior means and variances.
Crucially, these new objectives do not add any significant computational or memory overhead compared to standard MF-VI, making them easy "drop-in" replacements for existing BNN training pipelines.
Key Insights and Results
The authors evaluated their methods across a wide range of benchmarks, including UCI Regression datasets, image classification tasks (MNIST, CIFAR-10, etc.), and modern architectures like ResNet-18.
- Fixing Over-Pruning: The collapsed bound effectively "weakens" the regularization on the posterior means. This allows the model to shift the biases of pruned units to values that saturate activations (like ReLU), essentially "turning off" the noise from pruned weights rather than letting it leak into the final prediction.
- Tighter is Better for Tuning: The collapsed ELBO serves as a much more reliable signal for tuning hyperparameters (like the strength of the prior) than the standard ELBO. The peak of the collapsed ELBO aligns closely with the best test performance.
- MF-VI is More Capable Than We Thought: Contrary to the growing sentiment that Mean-Field VI cannot work for deep models, this paper demonstrates that treating priors correctly allows MF-VI to outperform many complex baselines.
Empirically, the results show a dramatic improvement over standard MF-VI:
- MNIST: CM-MF-VI achieved a test Negative Log-Likelihood (NLL) of 0.047, compared to 0.094 for standard MF-VI.
- CIFAR-10 (ResNet-18): The collapsed bound reduced the error rate to 13.34%, a massive leap over the 22.92% error seen with standard MF-VI.
- Out-of-Distribution (OOD) Robustness: Despite fitting the training data much better, the models retained their "Bayesian-ness." When evaluated on out-of-distribution data, the collapsed models correctly exhibited high predictive uncertainty.
Limitations and Future Work
While the method eliminates the need to tune prior parameters directly, it introduces "hyper-priors" (such as $\alpha_{reg}$). Fortunately, the authors demonstrate that these are much easier to tune because the ELBO itself acts as a reliable guide.
Additionally, the paper still operates within the mean-field framework, assuming weights are independent. Exploring how collapsed bounds interact with more expressive posterior distributions (like matrix-variate or low-rank Gaussians) is a promising next step. Finally, while scaling to ResNet-18 is a significant milestone, testing these bounds on massive transformer-based models remains an open challenge for the community.
Summary updated