Specifying Functional Priors in Bayesian Neural Networks

87 concepts65 papers1 proposal

Imagine you want to build a neural network to predict a patient's health metrics. As an expert, you have clear intuition about how this model should behave: its predictions should vary smoothly as a patient's vital signs change, and it should never output negative probabilities.

In a classical Bayesian framework, you would encode these beliefs into a prior distribution. However, because deep learning models are parameterized by weights and biases, you are forced to specify this prior over millions of abstract weight parameters—typically defaulting to a standard independent Gaussian distribution.

This highlights the fundamental disconnect in Bayesian deep learning: we have intuitive, human-understandable beliefs about functions (the outputs), but we are forced to express them as mathematical distributions over parameters (the weights). As Fortuin notes, this abstraction gap makes it nearly impossible to translate domain knowledge into parameter values, forcing practitioners to rely on simplistic default distributions that fail to reflect meaningful inductive biases.

Why the Parameter-to-Function Mapping Fails

Why can't we just use simple weight priors and trust the network to sort it out? The core issue is that the mapping from parameters to functions is highly non-linear, highly symmetric, and deeply pathological.

First, finding a probable set of weights does not mean you have found a probable function. As Wolpert demonstrated, the probability density shifts dramatically when moving through a non-linear mapping. Because multiple distinct weight configurations can represent the exact same input-output function, the volume of weight space warps the functional probability density, requiring a complex Jacobian correction term that standard estimation methods ignore.

Second, independent weight priors introduce severe functional pathologies as networks grow deeper:

  • Functional Collapse and Instability: Tran et al. and Wu et al. show that standard independent weight priors cause deep networks to exhibit pathological behavior in function space, frequently collapsing into overly simplistic constant functions or exploding into highly unstable, erratic curves.
  • Heavy-Tailed Outputs: Noci mathematically characterized the output distributions of finite-width ReLU networks under Gaussian weight priors. While wider networks lean toward predictable Gaussian behavior, deeper networks become increasingly heavy-tailed, eventually converging to a normal log-normal mixture that escapes intuitive control.
[Simple Weight Prior] ──(Highly Non-linear Mapping)──> [Pathological Function Space]
 (e.g., Independent Gaussian)                           (e.g., Collapsed or Heavy-Tailed)

Because the parameter space is vast and lacks a direct physical interpretation, researchers cannot easily predict how architectural choices or weight initializations fundamentally influence function-space behavior.

Empirical Consequences: Overconfidence and the Cold Posterior Effect

This disconnect isn't just a theoretical nuance; it actively undermines the primary reason we use Bayesian methods: reliable uncertainty quantification.

Standard Gaussian priors in high-dimensional neural networks often fail to represent true functional ignorance. Instead, Wilson points out that they frequently produce highly confident but incorrect predictions before any data is even observed. This lack of a principled uncertainty expansion prevents the model from safely flagging out-of-distribution inputs (Hafner et al.).

This mismatch is also the primary driver behind the Cold Posterior Effect, a well-known phenomenon where the theoretically "correct" Bayes posterior underperforms unless practitioners artificially "cool" or sharpen it ($T < 1$). As Noci et al. argue, standard Gaussian weight priors place excessive probability mass on overly complex functions. Artificially cooling the posterior acts as a heuristic fix to override the bad prior, forcing the network back toward simpler, more generalizable solutions.

Shifting the Paradigm to Function Space

To resolve these limitations, a major direction in modern machine learning seeks to bypass weight space entirely during prior specification and inference.

This idea traces its roots back to Williams et al., who proposed modeling the distribution of functions directly using Gaussian Processes (GPs), where properties like smoothness or periodicity are explicitly controlled via mathematical kernels. Modern functional inference attempts to bring this exact clarity to deep neural networks.

Traditional BNN:  Prior over Weights  ──> Complex Mapping ──> Inference over Weights
Functional BNN:   Prior over Functions ─────────────────────> Inference over Functions

Instead of trying to engineer uninterpretable weight distributions, frameworks like Functional Variational Inference (Ma et al.; Rudner et al.) optimize distributions directly over the network's outputs. By evaluating the model on a set of evaluation points, these methods calculate the divergence between the network's output distribution and a well-behaved functional prior (like a GP).

To avoid the mathematical pathologies of comparing infinite-dimensional spaces, the authors often leverage alternative probability metrics, such as the Wasserstein distance (Wild et al.), to smoothly align the neural network's behavior with desired functional properties.

By shifting the target of our priors from what the model is (its weights) to what the model does (its outputs), practitioners can directly inject explicit domain knowledge—such as high-level summary statistics (Raj et al.), interpretable safety constraints (Yang et al.), or controlled uncertainty bounds in unexplored regions of the feature space (Klarner et al.).

Key Foundational Work

Gaussian Processes for Regression (1995) — contributed 1 concept elements to this direction.
Bayesian Backpropagation Over I-O Functions Rather Than Weights (1993) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 66 papers
  2. Version 1 · 2026-10-04 · 119 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_weight_space_prior_limitations_bd7301d4,
  title = {weight-space prior limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weight-space-prior-limitations/rdr_bd7301d45fc44e05a9acda5094306ada},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_weight_space_prior_limitations_bd7301d4_v2,
  title = {weight-space prior limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weight-space-prior-limitations/rdr_bd7301d45fc44e05a9acda5094306ada/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake