Introduction
Bayesian inference in the space of functions has long been a sought-after goal in Bayesian modeling. This paper introduces Functional Variational Inference (FVI), a novel framework for performing Bayesian inference directly in the space of functions rather than neural network weights. By utilizing a new class of flexible distributions called Stochastic Process Generators (SPGs) and a "grid-functional" objective, FVI achieves superior uncertainty quantification and predictive performance while operating significantly faster than previous function-space methods.
The Challenge of Weight-Space Inference
Traditional Bayesian Neural Networks (BNNs) typically perform inference in weight space. This approach presents two primary challenges:
- Identifiability: Multiple distinct weight configurations can produce the exact same function. This many-to-one mapping creates a highly complex, multi-modal posterior that is notoriously difficult to sample or approximate.
- Prior Intuition: It is highly unintuitive to map a prior distribution over weights (e.g., assuming weights are normally distributed) to the resulting function's behavioral characteristics.
To address these issues, researchers have explored function-space inference, defining priors and performing updates directly on the functions the model produces. However, existing methods like functional BNNs (f-BNNs) and Variational Implicit Processes (VIPs) face significant hurdles. Their mathematical objectives, such as the KL divergence, are often ill-defined or infinite. Furthermore, they frequently rely on slow gradient estimators or are restricted to simple Gaussian approximations incapable of capturing complex patterns.
Proposed Methodology: FVI and SPGs
The authors propose two major technical innovations to make function-space inference both practical and mathematically sound:
1. The Grid-Functional KL Divergence
Standard functional KL divergence is defined as the supremum over all possible measurement points, which often leads to infinite values when comparing different network architectures. FVI circumvents this by introducing a "grid-functional" approach. It takes an expectation over a random number of measurement points (the grid). This ensures the objective remains finite, mathematically valid, and computationally tractable.
2. Stochastic Process Generators (SPGs)
SPGs represent a new variational family acting as a generator for functions—conceptually similar to a Variational Autoencoder (VAE) but for stochastic processes. An SPG uses a set of basis functions (parameterized by deep networks) whose coefficients are determined by a latent variable. This architecture allows the model to represent highly non-Gaussian distributions of functions, offering a significant upgrade over the restrictive Gaussian Process approximations used in prior work.
The FVI algorithm operates in two main steps: first, it "distills" the desired prior (such as a BNN prior) into an SPG. Second, it optimizes a variational SPG to match the posterior by maximizing a functional Evidence Lower Bound (ELBO). An unbiased "Russian Roulette" estimator is employed to ensure stable mini-batch sampling.
Experiments and Key Results
The authors evaluated FVI across a diverse range of benchmarks, including synthetic 1D regression, structured implicit priors, UCI regression datasets, contextual bandits, and image classification with Out-of-Distribution (OOD) detection.
Improved Uncertainty Quantification: In 1D regression tests, FVI successfully modeled "in-between" uncertainty—the epistemic uncertainty in the gap between data clusters. Standard mean-field BNNs typically fail completely in these regions, exhibiting overconfidence.
Predictive Performance and Decision Making: FVI outperformed weight-space VI and other function-space baselines on 7 out of 9 UCI regression tasks. In contextual bandits, FVI ranked first overall in cumulative regret, demonstrating that its uncertainty estimates are reliable for real-world exploration-exploitation trade-offs.
Scalability and Efficiency: By utilizing a VAE-like latent structure, FVI avoids the $O(M^3)$ scaling of Stein gradient estimators used in f-BNNs. FVI scales linearly with the number of function samples, making it 100x to 500x faster in practice. This efficiency allows FVI to achieve state-of-the-art OOD detection on high-dimensional image datasets like FashionMNIST and CIFAR-10, where previous functional methods struggled to scale.
Limitations and Future Work
While FVI represents a significant step forward, it currently relies on a two-step process where the prior must first be distilled into a surrogate SPG; poor distillation can negatively impact final inference. Additionally, while the grid-functional KL is a valid divergence, its theoretical properties compared to standard functional KL divergence warrant further investigation. Future work may focus on estimating the grid-functional KL directly without the need for a surrogate model, streamlining the inference pipeline.
Summary updated