Function Space Priors for Bayesian Deep Learning

24 concepts19 papers

When building a Bayesian Neural Network (BNN), our goal is to capture uncertainty by placing a prior distribution over the model's parameters. Traditionally, practitioners choose a simple independent Gaussian distribution over the weights: $w \sim \mathcal{N}(0, \sigma^2 I)$.

However, this choice suffers from a fundamental pathology: humans do not have intuition about weights, we have intuition about functions. A deep neural network is a highly non-linear, overparameterized mapping. As reviewed by Fortuin, placing a "vague" default prior on individual weights results in highly unpredictable, chaotic, or degenerate distributions in the actual output space—the function space where the model makes predictions.

To fix this, a growing body of research shifts the perspective entirely: instead of defining what the weights should look like, we define how the function should behave. By specifying priors directly in function space, we can explicitly encode structured inductive biases like smoothness, periodicity, or spatial invariance.


The Weight-Space Pathology

To understand why function-space priors are necessary, consider what happens when you initialize a standard BNN. If you sample a set of weights from a standard Gaussian prior, the resulting function often behaves erratically or defaults to an overly simplistic, flat line when extrapolating beyond the training data. As Tran et al. demonstrate, weight-space priors fail to guarantee that the network will preserve basic behavioral properties like reasonable lengthscales or smooth transitions.

This disconnect occurs because the relationship between a network's weights and its output function is deeply complex. A single functional behavior can be represented by infinitely many different weight configurations. Conversely, a tiny modification in weight space can catastrophically alter the function's global behavior.

Ideally, we want our models to inherit the elegant properties of non-parametric methods like Gaussian Processes (GPs), which allow us to explicitly dictate functional behavior via a kernel. Sun et al. formalized this shift by introducing Functional Variational BNNs (fBNNs), arguing that modeling uncertainty in the space of the functions the networks actually compute makes the entire Bayesian framework more interpretable and robust.


How Function-Space Priors Work

We cannot easily write down an analytical probability density function for the outputs of a deep neural network. How, then, do we force a neural network to conform to a function-space prior?

The core mechanism relies on measurement points (or context points). While we cannot evaluate the function's distribution across the entire infinite input domain, we can evaluate it at a finite set of representative inputs, $X = \{x_1, x_2, \dots, x_N\}$.

  +---------------------------------------------------------+
  |                  Function-Space Prior                   |
  |         (e.g., Gaussian Process / Simulator)            |
  +----------------------------+----------------------------+
                               |
            Evaluate at Measurement Points (X)
                               |
                               v
  +---------------------------------------------------------+
  |               Target Distribution p(f_X)                |
  +----------------------------+----------------------------+
                               |
                       Minimize Distance
             (KL Divergence / Wasserstein Distance)
                               |
                               v
  +---------------------------------------------------------+
  |             Induced BNN Distribution q_θ(f_X)           |
  +----------------------------+----------------------------+
                               ^
            Evaluate at Measurement Points (X)
                               |
  +----------------------------+----------------------------+
  |                 Bayesian Neural Network                 |
  |              with Parameter Distribution q(w)           |
  +---------------------------------------------------------+
  1. Define a Reference Prior: We choose a stochastic process that embodies our inductive biases. For instance, Bamler et al. utilize Gaussian Processes to specify properties like periodicity or smoothness.
  2. Sample Measurement Points: We select a set of input locations $X$. These can include your training data, but crucially, they must also include points outside your training distribution to constrain how the model extrapolates.
  3. Align the Distributions: We evaluate the distribution of the prior at these points, $p(f_X)$, and the distribution of the BNN's outputs at these same points, $q_\theta(f_X)$. We then optimize the BNN's parameters to minimize a distance metric between these two distributions.

For example, Tran et al. use Wasserstein distance optimization to "tune" weight distributions until their functional outputs closely mimic a target GP. To ensure mathematical consistency across infinite-dimensional function spaces, Bamler et al. developed a regularized KL-divergence framework that prevents the optimization objective from becoming ill-defined or exploding.

Beyond Gaussian Processes: Implicit Processes

What if our prior knowledge cannot be captured by a standard GP kernel? Ma et al. expanded this paradigm by introducing Variational Implicit Processes (VIP). Instead of relying on an explicit analytical prior, VIP allows researchers to use implicit processes—such as simulators or secondary generative neural networks—as functional priors, combining the flexible representational power of deep learning with principled uncertainty quantification.


Algorithmic Paradigms

Translating this intuition into scalable algorithms has led to several distinct inference frameworks that operate directly in function space.

Particle and Gradient Methods

Instead of maintaining an explicit parametric distribution over weights, we can maintain a collection of functional "particles." Pielok et al. developed Stein Functional Variational Gradient Descent (SFVGD), which updates neural network ensembles by driving their outputs along a functional gradient. This approach pushes the particles to fit the training data while simultaneously applying a functional force that pulls them toward the GP prior in unobserved regions.

Functional MCMC

For highly complex, multi-modal distributions, variational approximations can fall short. To address this, Wu et al. and Wu et al. developed functional Stochastic Gradient MCMC frameworks. By lifting diffusion dynamics from abstract parameter spaces directly into function space, these methods allow the sampler to navigate the landscape of actual model behaviors, capturing diverse functional modes that traditional weight-space samplers routinely miss.

Calibrating Ensembles and Laplace Approximations

The utility of function-space priors extends beyond pure BNNs. For instance, deep ensembles are highly popular but prone to overconfidence out-of-distribution. Deng et al. introduced a method that treats a deep ensemble as a Gaussian Process, using functional variational inference to anchor the ensemble's uncertainty envelope. Similarly, Weber et al. integrated function-space priors into Laplace approximations, allowing standard deterministic networks to be retrofitted with reliable extrapolation behavior.


Why It Works: Out-of-Distribution Anchoring

The ultimate value of a function-space prior becomes obvious when a model encounters out-of-distribution (OOD) data.

Inside the training data distribution, the data likelihood dominates the objective function. Both weight-space and function-space models will force the network to fit the data points. However, in regions far away from the training data, the likelihood provides no guidance.

A standard BNN or deep ensemble has unconstrained functional behavior in these regions; its parameters can combine to produce highly confident, wildly incorrect predictions. As Hafner et al. demonstrate with Noise Contrastive Priors (NCPs), we can combat this by training models on "fantasized" OOD noise inputs, explicitly forcing the model's output variance to widen.

By evaluating a function-space prior on these OOD measurement points, the optimization objective acts as an anchor. When the model extrapolates into empty space, the functional prior forces it to match the reference behavior—meaning the network naturally reverts to a baseline state or expresses a wide, realistic uncertainty envelope rather than overconfident delusions.

Training Data Region                     Out-of-Distribution Region
====================                     ==========================
Data Likelihood Dominates                Functional Prior Dominates
Model fits the data points               Model matches prior behavior (e.g., GP)
Low Uncertainty                          High Uncertainty / Smooth Extrapolation

Applications Across Domains

The philosophy of encoding domain knowledge directly into the functional or geometric space of a problem has proven powerful across several specialized fields:

  • Kernel Learning: Instead of manually engineering fixed parametric kernels for GPs, Benton et al. applied function-space distributions to the kernel itself, placing a prior over the kernel's spectral density to let the model automatically discover complex structures.
  • Probabilistic Numerics: In numerical analysis, mathematical operations like integration can be framed as inference tasks. Briol et al. showed that encoding structural properties like positivity or boundary conditions directly into the functional prior drastically improves sample efficiency and error propagation.
  • Reinforcement Learning: Standard model-based RL often wastes capacity trying to learn pixel-perfect world simulators. Farahmand et al. introduced Value-Aware Model Learning (VAML), which restricts the model-learning loss to specific functional spaces (like Sobolev or RKHS spaces) relevant to the agent's value function, ignoring high-frequency environmental noise that does not impact decision-making.
  • Time-Series Forecasting: To handle severe temporal distribution shifts, Wang et al. incorporated a dictionary of predefined mathematical functions (such as sines and polynomials) as a functional inductive bias, allowing models to robustly maintain periodic and growth-oriented patterns even when the underlying data distribution shifts.

Key Foundational Work

Functional variational Bayesian Neural Networks (2019) — contributed 1 concept elements to this direction.
Function-Space Distributions over Kernels. (2019) — contributed 1 concept elements to this direction.
Variational Implicit Processes (2019) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 19 papers
  2. Version 1 · 2026-10-04 · 19 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_functional_inductive_bias_7467f062,
  title = {functional inductive bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/functional-inductive-bias/rdr_7467f06263f04573a7c10cd06752cc30},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_functional_inductive_bias_7467f062_v2,
  title = {functional inductive bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/functional-inductive-bias/rdr_7467f06263f04573a7c10cd06752cc30/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake