Function Space Priors for Bayesian Neural Networks

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2119 papers141 conceptsfrozen 2026-10-04

Bayesian Deep Learning (BDL) promises a principled way to handle uncertainty: instead of learning a single set of weights, we learn a distribution over all possible weights. However, as Vincent Fortuin notes in Priors in Bayesian Deep Learning: A Review, the field faces an "identity crisis." While we care about the model's behavior—the function it represents—we traditionally define our prior beliefs in the weight space.

This disconnect creates a fundamental problem: it is nearly impossible for a human expert to translate an intuitive belief (like "the function should be smooth") into a specific distribution over millions of abstract, high-dimensional parameters.

The Weight Space Paradox

In a standard Bayesian Neural Network (BNN), we might place a simple Gaussian prior on the weights. But as Amir Emad Marvasti and colleagues argue in Out of Distribution Detection, Generalization, and Robustness Triangle with Maximum Probability Theorem, these choices are often arbitrary. Because the relationship between weights and outputs is highly non-linear and opaque, a "simple" weight prior can induce incredibly complex, heavy-tailed, and undesirable behaviors in the actual function the network computes.

Furthermore, the weight space is redundant. Hrittik Roy demonstrates in Reparameterization invariance in approximate Bayesian inference that many different weight configurations can represent the exact same mathematical function. Standard Bayesian methods often waste "probability mass" on these redundant configurations, leading to unreliable uncertainty estimates that don't reflect the model's actual diversity of reasoning.

Moving to Function Space

To solve this, researchers are shifting the focus from what the weights are to what the function does. This is known as function-space inference.

The intuition is simple: instead of regularizing weights, we regularize the model's predictions. In Function-Space Regularization for Deep Bayesian Classification, Jihao Andreas Lin and co-authors propose placing priors directly on the model's outputs. This allows practitioners to specify desired characteristics—like how a model should behave in regions where there is no training data—without ever needing to interpret the underlying weights.

This approach is particularly powerful for incorporating expert knowledge. Wanqian Yang and Lars Lorch introduced Output-Constrained Bayesian Neural Networks, which allow researchers to bake "common sense" or safety rules directly into the prior. For example, in a medical application, an expert can specify that a certain output must never exceed a safety threshold, and the function-space prior will enforce this across the entire distribution of learned models.

Bridging Neural Networks and Gaussian Processes

The gold standard for function-space modeling has long been the Gaussian Process (GP), which uses kernels to define properties like smoothness or periodicity directly. However, GPs struggle to scale to the massive datasets where neural networks thrive.

Recent work seeks to build a mathematical bridge between these two worlds. Aidan Scannell and colleagues developed Sparse Function-space Representation (SFR), which converts trained neural networks into sparse GPs. This gives the network the "superpowers" of a GP—principled uncertainty and the ability to learn sequentially without forgetting—while keeping the scalability of deep learning.

Similarly, Tim G. J. Rudner and Yarin Gal have pushed for Tractable Function-Space Variational Inference. Their work aims to make it computationally feasible to align a neural network's behavior with a specific functional prior, such as ensuring the model remains "uncertain" in unexplored areas of the feature space—a critical requirement for safety-critical AI.

Learning the Prior from Data

If we can't easily design a prior, perhaps we can learn one. Tommy Rochussen explores this in Amortising Inference and Meta-Learning Priors in Neural Networks, where the weight priors are meta-learned directly from data. By treating the BNN as part of a larger stochastic process, the model learns what a "reasonable" function looks like for a given task, moving away from the simplistic Gaussian defaults that Vincent Fortuin identified as a primary bottleneck for the field.

By centering the function rather than the parameter, this research direction is turning Bayesian Neural Networks from abstract mathematical constructs into interpretable, controllable tools for reliable machine learning.

Papers in this version

Showing 60 of 119 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 66 papers
  2. Version 1 · 2026-10-04 · 119 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_weight_space_prior_limitations_bd7301d4,
  title = {weight-space prior limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weight-space-prior-limitations/rdr_bd7301d45fc44e05a9acda5094306ada},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_function_space_priors_for_bayesian_neura_bd7301d4_v1,
  title = {Function Space Priors for Bayesian Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/function-space-priors-for-bayesian-neural-networks/rdr_bd7301d45fc44e05a9acda5094306ada/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}