Abstract
Encoding domain knowledge into the prior over the high-dimensional weight space of a neural network is challenging but essential in applications with limited data and weak signals. Two types of domain knowledge are commonly available in scientific applications: feature sparsity and signal-to-noise ratio (quantified as the proportion of variance explained, PVE). We propose a new joint prior over local scale parameters to encode feature sparsity and a Stein gradient optimization to tune hyperparameters such that the induced PVE distribution matches a prior belief. Empirically, this improves prediction accuracy in weak-signal regimes, such as genetics, often outperforming computationally intensive cross-validation.
Introduction
Deep learning typically requires massive datasets with high signal-to-noise ratios. However, scientific fields like genetics often deal with "weak signal" regimes: thousands of features, small sample sizes, and signals easily obscured by noise. While Bayesian Neural Networks (BNNs) offer a framework for regularization via priors, standard Gaussian priors are often "uninformative" and fail to capture domain-specific knowledge. For example, a researcher might know that only a small fraction of features are relevant (sparsity) and that these features explain a specific portion of the variance (PVE). Translating these functional beliefs into high-dimensional weight distributions is a significant challenge. This paper introduces a method to bake these specific types of domain knowledge directly into BNN priors.
Methodology
Informative Spike-and-Slab Prior
Traditional spike-and-slab priors treat each feature independently. The authors propose a joint prior over feature inclusion. By placing a distribution (e.g., a "flattened" Laplace) on the total number of relevant features, the model creates a dependency: identifying several strong features makes the prior more skeptical of others. This effectively budgets relevance and prunes false positives that appear significant due to random noise.
PVE Matching via Stein Gradients
The Proportion of Variance Explained (PVE) is a functional property of the network's output. To align weight-space priors with functional-space beliefs, the authors use Stein Gradient Estimators (SGE). They optimize the hyperparameters of the weight priors so the distribution of the PVE induced by the network matches a pre-defined prior belief (like a Beta distribution). This ensures the network "expects" the correct signal level before training begins.
Empirical Evaluation
The approach was validated across synthetic data, UCI regression benchmarks (with added noise features), web traffic forecasting, and FINRISK genetics data. In the genetics application, predicting metabolite levels from high-dimensional SNP data represents a classic weak-signal problem.
Key Results
- Noise Resilience: On high-noise synthetic data, the proposed "InfoSS" prior significantly outperformed standard spike-and-slab and Horseshoe priors.
- Feature Selection: In UCI datasets augmented with 100 irrelevant features, informative priors consistently achieved lower Mean Squared Error (MSE) by effectively filtering noise.
- Genetics Performance: Incorporating PVE knowledge improved accuracy in metabolite prediction. For the gene APOA5, the informative BNN outperformed both Lasso and standard BNNs.
Discussion and Conclusion
A key insight is that dependency in feature priors—moving from independent counts to a joint budget—is highly effective for noise filtering in high dimensions. Furthermore, the PVE matching technique provides a scalable way to make weight-space priors behave like functional-space priors. In many instances, encoding these "rough guesses" outperformed computationally expensive cross-validation, suggesting that expert intuition is an underutilized but powerful regularizer.
Future work includes extending PVE definitions to classification tasks and exploring the combination of these priors with MCMC or Deep Ensembles to improve uncertainty quantification, as Variational Inference may underestimate posterior variance.
Summary updated