Informative Gaussian Scale Mixture Priors for Bayesian Neural Networks

Tianyu Cui, Aki S. Havulinna, Pekka Marttinen, Samuel Kaski

Introduction

Bayesian Neural Networks (BNNs) offer a principled framework for uncertainty quantification, yet they often struggle in "weak signal" regimes common in scientific fields like genetics. In these domains, the signal-to-noise ratio is low, and the underlying structure is highly sparse—only a few features out of thousands may be relevant. Standard BNN priors are typically non-informative and fail to incorporate high-level domain knowledge, such as the expected number of relevant features or the total variance the model should explain. This paper introduces a framework to embed such domain expertise directly into Gaussian Scale Mixture (GSM) priors.

Informative Spike-and-Slab

The first innovation addresses feature sparsity. Traditional spike-and-slab priors assume independent feature inclusion, which often fails to suppress noise effectively in high-dimensional settings. The authors propose an Informative Spike-and-Slab prior that defines a joint distribution over the number of relevant features.

By using a "flattened Laplace" or similar distribution to express a range of likely feature counts, the model makes feature inclusion dependent. Once the model identifies a set of "true" features that satisfy the prior's count, it becomes significantly more aggressive at penalizing remaining features as noise. This collective behavior allows the BNN to distinguish subtle signals from background noise more effectively than independent Bernoulli trials.

Comparison of standard and informative spike-and-slab priors on feature selection and weight estimation.

PVE Matching via Stein Gradients

The second innovation bridges the gap between uninterpretable weight scales and scientific intuition. Researchers often have a prior belief about the Proportion of Variance Explained (PVE)—essentially the $R^2$ of the model. However, translating a desired PVE (e.g., a Beta distribution) into specific hyperparameter settings for weight priors is non-trivial because the PVE is an implicit property of the network's architecture and weights.

The authors utilize the Stein Gradient Estimator (SGE) to optimize the hyperparameters of the GSM priors. This process ensures that the distribution of functions sampled from the BNN prior matches the target PVE distribution. This "PVE matching" provides a universal language for scientists to guide deep learning models using established domain metrics.

Optimization of PVE distribution and its effect on function samples from the BNN prior.

Experimental Results

The framework was rigorously tested across synthetic data, augmented UCI benchmarks, and real-world genetic data (FINRISK).

  • Robustness to Noise: On synthetic tasks with high noise levels, the Informative Spike-and-Slab (InfoSS) consistently outperformed standard sparsity-inducing priors like the Horseshoe.
  • High-Dimensional Benchmarks: On UCI datasets modified with 100 irrelevant features, the combination of sparsity and PVE knowledge (InfoHMF+PVE) achieved top performance in the majority of cases, demonstrating its ability to filter "junk" features.
  • Genetics Application: In the FINRISK study, the informative approach was the only method to achieve a positive test PVE for the difficult APOA5 gene. Other baselines, including Lasso and standard BNNs, overfitted to the point of producing negative results, highlighting the necessity of informative priors in extreme low-signal environments.

Conclusion

This work demonstrates that "uninterpretable" neural networks can be effectively guided by high-level domain knowledge. By anchoring BNN priors to physical and statistical properties like sparsity and PVE, the authors provide a robust tool for scientific discovery where data is noisy and signals are weak. This approach not only improves predictive accuracy but also reduces the reliance on computationally expensive and high-variance hyperparameter tuning methods like cross-validation.

Summary updated

BibTeX
@inproceedings{cui2021informative,
  author = {Tianyu Cui and Aki S. Havulinna and Pekka Marttinen and Samuel Kaski},
  title = {{Informative Gaussian Scale Mixture Priors for Bayesian Neural Networks}},
  booktitle = {Bayesian},
  year = {2021},
  doi = {10.1214/21-ba1291},
}
Figure