Structured Partial Stochasticity in Bayesian Neural Networks

Tommy Rochussen

TL;DR

This paper tackles the challenge of multi-modal posteriors in Bayesian Neural Networks (BNNs) caused by weight symmetries. The authors propose a novel method of "structured partial stochasticity" by selectively fixing certain weights to break these symmetries, leading to a simplified posterior distribution and improved performance for approximate inference methods.

What Problem Does This Paper Address?

Bayesian Neural Networks (BNNs) offer a powerful framework for uncertainty estimation, but their practical application is often hindered by the complexity of their posterior distributions. A significant issue arises from symmetries in neural network architectures, particularly neuron permutation symmetries. These symmetries mean that different sets of weights can result in the exact same network function. When combined with standard isotropic priors, this leads to BNN posteriors with a factorial number of identical modes, making approximate inference (like Variational Inference) difficult and inefficient. While previous work has explored partial stochasticity (making only a subset of weights stochastic), this paper aims to systematically break these problematic symmetries to simplify the posterior and improve inference.

Methods

The core contribution is a method called structured partial stochasticity. Instead of making all weights stochastic, the authors propose to fix a subset of weights as deterministic. This fixing is done in a structured way to specifically target and break neuron permutation symmetries.

Two main schemes are introduced:

  • Light Scheme: This involves removing connections between the "top" neurons of adjacent layers and the "bottom" neurons of adjacent layers. This is equivalent to applying a mask to the weight matrices with zeros on specific diagonal elements.
  • Heavy Scheme: This scheme is more aggressive. It removes connections not just from the top/bottom neurons but also from the second top/bottom, third top/bottom, and so on. This ensures that even if some non-fixed weights are close to zero, the symmetry is broken.

The fixed weights can be set to:

  1. A constant value c or -c with equal probability.
  2. Their corresponding Maximum A Posteriori (MAP) weights.

By fixing these weights, the authors aim to reduce the number of redundant modes in the posterior, making approximate inference more tractable and accurate.

Datasets

  • 1D artificial regression dataset: Used for initial evaluation of the proposed methods.
  • UCI Energy Efficiency dataset: A standard regression dataset used for evaluating the performance of the proposed schemes on a more complex, real-world task.

Key Insights and Ideas

  • Symmetries are a major bottleneck for BNN inference: The paper strongly argues that the multi-modal nature of BNN posteriors due to symmetries is a primary reason for the limitations of current approximate inference techniques.
  • Structured weight fixing is key: Simply making weights random (unstructured partial stochasticity) doesn't solve the symmetry problem as effectively as a structured approach that specifically targets the symmetry-breaking.
  • "Heavy" pruning is more robust: The heavy scheme, which fixes more weights, is shown to be more effective at breaking symmetries and leading to better inference results, especially when non-fixed weights might be close to zero.
  • Simplifying the posterior directly improves inference: By reducing the number of modes, the paper demonstrates that approximate inference methods can achieve better predictive performance and uncertainty estimates.

Key Results

  • Improved Predictive Performance: Across both toy regression and the UCI Energy Efficiency dataset, the proposed structured partial stochasticity methods (especially the "heavy" variants) consistently outperformed vanilla MFVI (Mean-Field Variational Inference).
  • Better Uncertainty Estimates: The heavy fixing with c=5 on the toy regression task showed "impressive in-between uncertainty," suggesting improved calibration.
  • Symmetry-breaking is crucial: The paper highlights that unstructured random fixing of weights (HRF, LRF) performed poorly compared to the structured schemes, underscoring the importance of the symmetry-breaking approach. For instance, on the UCI Energy Efficiency dataset, RMSE was significantly lower and LPP was higher for structured methods compared to random fixing.

Limitations and Future Work

  • Limited Scope: The current work focuses on fully-connected feed-forward networks and primarily regression tasks.
  • Adverse Effects on True Posterior: The paper acknowledges the need to investigate if the proposed symmetry-removal schemes have any adverse effects on the true posterior distribution.
  • Generalization to Other Architectures and Inference Methods: Future work aims to refine these schemes for more sophisticated architectures (e.g., CNNs, RNNs) and explore their application with other approximate inference techniques beyond MFVI.
  • Investigating the Impact on True Posterior: A crucial future direction is to understand if these symmetry-breaking techniques inadvertently distort the true posterior.

Summary updated

Abstract

Bayesian neural network posterior distributions have a great number of modes that correspond to the same network function. The abundance of such modes can make it difficult for approximate inference methods to do their job. Recent work has demonstrated the benefits of partial stochasticity for approximate inference in Bayesian neural networks; inference can be less costly and performance can sometimes be improved. I propose a structured way to select the deterministic subset of weights that removes neuron permutation symmetries, and therefore the corresponding redundant posterior modes. With a drastically simplified posterior distribution, the performance of existing approximate inference schemes is found to be greatly improved.
BibTeX
@inproceedings{rochussen2024structured,
  author = {Tommy Rochussen},
  title = {{Structured Partial Stochasticity in Bayesian Neural Networks}},
  booktitle = {AABI},
  year = {2024},
  url = {https://openreview.net/forum?id=sS4dQOmCk0},
}
Figure