Introduction
Deep learning models often exhibit overconfidence, which poses significant risks in high-stakes applications such as healthcare and autonomous driving. Bayesian Neural Networks (BNNs) offer a solution by placing probability distributions over network weights to quantify uncertainty. However, performing inference in the high-dimensional, highly correlated weight space of modern networks is notoriously difficult. Standard techniques like Markov Chain Monte Carlo (MCMC) struggle with mixing, while Variational Inference (VI) often falls into local optima or relies on simplistic "mean-field" approximations that ignore crucial weight correlations.
This paper introduces Proj-BNN, a framework that learns a non-linear projection from a low-dimensional latent space to the full network weight space. By performing variational inference in this simplified latent space, Proj-BNN bypasses high-dimensional optimization pathologies and successfully captures complex, multimodal posteriors.
Methodology: The Proj-BNN Framework
The central hypothesis of Proj-BNN is that while high-dimensional spaces are suitable for learning point-estimate parameters, characterizing Bayesian uncertainty is more effective in a lower-dimensional latent space. The authors propose a three-stage inference process:
- Weight Characterization: The method uses Fast Geometric Ensembling (FGE) to sample a diverse set of high-quality network weights (MAP solutions).
- Prediction-Constrained Projection: A novel autoencoder is trained on these weights. Unlike standard autoencoders, the loss function includes a penalty for drops in predictive accuracy of the reconstructed weights, ensuring the latent space preserves functional utility.
- Latent Variational Inference: Using the autoencoder's decoder as a projection function, Black-Box Variational Inference (BBVI) is applied to learn the posterior distribution over the low-dimensional latent variables.
This approach allows a simple, fully factorized distribution in the latent space to map to a highly complex, correlated distribution in the actual weight space, effectively capturing multiple modes of plausible weights as seen in the PCA visualization of weight clusters.
Experimental Results
The authors evaluated Proj-BNN on synthetic 1D regression tasks and eight standard UCI benchmark datasets.
Qualitative Uncertainty and Multimodality
On synthetic datasets designed with complex, multimodal posteriors, Proj-BNN demonstrated a superior ability to capture multiple functional modes. In contrast, baseline methods like Bayes by Backprop (BbB) typically collapse to a single mode, leading to overconfident predictions in regions with sparse data.
Quantitative Performance
On the UCI benchmarks, Proj-BNN achieved competitive or superior test log-likelihood compared to state-of-the-art structured VI techniques, including Multiplicative Normalizing Flows (MNF) and Matrix Variate Gaussian (MVG) posteriors. Furthermore, Proj-BNN showed greater robustness in extrapolation and interpolation tests, maintaining reliable uncertainty estimates where other methods failed.
Discussion and Future Work
While Proj-BNN significantly improves uncertainty quantification, its performance is tied to the quality of the initial weight sampling (FGE). If the initial samples do not adequately explore the weight space, the latent projection will be limited. Future research could focus on training objectives that explicitly encourage diversity during the initial sampling stage. Additionally, the latent projection framework shows promise for multi-task and meta-learning, where shared projection parameters can allow for statistical strength across different functions.
Summary updated