Latent Projection BNNs: Avoiding weight-space pathologies by learning latent representations of neural network weights

Melanie F. Pradier, Weiwei Pan, Jiayu Yao, Soumya Ghosh, Finale Doshi-VelezarXiv 2018 · arXiv

Introduction

Deep learning models often exhibit overconfidence, which poses significant risks in high-stakes applications such as healthcare and autonomous driving. Bayesian Neural Networks (BNNs) offer a solution by placing probability distributions over network weights to quantify uncertainty. However, performing inference in the high-dimensional, highly correlated weight space of modern networks is notoriously difficult. Standard techniques like Markov Chain Monte Carlo (MCMC) struggle with mixing, while Variational Inference (VI) often falls into local optima or relies on simplistic "mean-field" approximations that ignore crucial weight correlations.

This paper introduces Proj-BNN, a framework that learns a non-linear projection from a low-dimensional latent space to the full network weight space. By performing variational inference in this simplified latent space, Proj-BNN bypasses high-dimensional optimization pathologies and successfully captures complex, multimodal posteriors.

Methodology: The Proj-BNN Framework

The central hypothesis of Proj-BNN is that while high-dimensional spaces are suitable for learning point-estimate parameters, characterizing Bayesian uncertainty is more effective in a lower-dimensional latent space. The authors propose a three-stage inference process:

  1. Weight Characterization: The method uses Fast Geometric Ensembling (FGE) to sample a diverse set of high-quality network weights (MAP solutions).
  2. Prediction-Constrained Projection: A novel autoencoder is trained on these weights. Unlike standard autoencoders, the loss function includes a penalty for drops in predictive accuracy of the reconstructed weights, ensuring the latent space preserves functional utility.
  3. Latent Variational Inference: Using the autoencoder's decoder as a projection function, Black-Box Variational Inference (BBVI) is applied to learn the posterior distribution over the low-dimensional latent variables.
PCA visualization of the learned latent weight space showing distinct clusters.

This approach allows a simple, fully factorized distribution in the latent space to map to a highly complex, correlated distribution in the actual weight space, effectively capturing multiple modes of plausible weights as seen in the PCA visualization of weight clusters.

Experimental Results

The authors evaluated Proj-BNN on synthetic 1D regression tasks and eight standard UCI benchmark datasets.

Qualitative Uncertainty and Multimodality

On synthetic datasets designed with complex, multimodal posteriors, Proj-BNN demonstrated a superior ability to capture multiple functional modes. In contrast, baseline methods like Bayes by Backprop (BbB) typically collapse to a single mode, leading to overconfident predictions in regions with sparse data.

Comparison of predictive uncertainty across different BNN methods on a toy regression task. Proj-BNN (a) captures broader uncertainty compared to BbB (b), MNF (c), and MVG (d).

Quantitative Performance

On the UCI benchmarks, Proj-BNN achieved competitive or superior test log-likelihood compared to state-of-the-art structured VI techniques, including Multiplicative Normalizing Flows (MNF) and Matrix Variate Gaussian (MVG) posteriors. Furthermore, Proj-BNN showed greater robustness in extrapolation and interpolation tests, maintaining reliable uncertainty estimates where other methods failed.

Discussion and Future Work

While Proj-BNN significantly improves uncertainty quantification, its performance is tied to the quality of the initial weight sampling (FGE). If the initial samples do not adequately explore the weight space, the latent projection will be limited. Future research could focus on training objectives that explicitly encourage diversity during the initial sampling stage. Additionally, the latent projection framework shows promise for multi-task and meta-learning, where shared projection parameters can allow for statistical strength across different functions.

Summary updated

Abstract

As machine learning systems get widely adopted for high-stake decisions, quantifying uncertainty over predictions becomes crucial. While modern neural networks are making remarkable gains in terms of predictive accuracy, characterizing uncertainty over the parameters of these models is challenging because of the high dimensionality and complex correlations of the network parameter space. This paper introduces a novel variational inference framework for Bayesian neural networks that (1) encodes complex distributions in high-dimensional parameter space with representations in a low-dimensional latent space, and (2) performs inference efficiently on the low-dimensional representations. Across a large array of synthetic and real-world datasets, we show that our method improves uncertainty characterization and model generalization when compared with methods that work directly in the parameter space.
BibTeX
@misc{pradier2018latent,
  author = {Melanie F. Pradier and Weiwei Pan and Jiayu Yao and Soumya Ghosh and Finale Doshi-Velez},
  title = {{Latent Projection BNNs: Avoiding weight-space pathologies by learning latent representations of neural network weights}},
  year = {2018},
  eprint = {1811.07006},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.07006},
}
Figure