Standard deep neural networks are notoriously overconfident. When presented with out-of-distribution (OOD) data or unfamiliar scenarios, they routinely output highly confident, incorrect predictions. Traditionally, researchers have tried to mitigate this by regularizing the network's parameters—either through weight decay or by placing prior distributions over the weights in a Bayesian Neural Network (BNN).
However, this introduces a fundamental mismatch: we regularize in parameter space, but we only care about the model's behavior in function space (the mapping from inputs to outputs). Shifting the regularization target directly to the function space offers a more intuitive, architecture-agnostic, and effective way to control model behavior and quantify uncertainty.
The Parameter-Space Paradox
To understand why function-space regularization works, we must first look at why parameter-space regularization fails.
Neural networks exhibit extreme weight-space symmetry. Because you can permute the hidden units or invert weights in adjacent layers without changing the network's outputs, millions of distinct parameter configurations map to the exact same underlying function. As Badino et al. (2017) note, forcing internal weights to align with a central reference model can be counterproductive because it ignores these symmetries.
Furthermore, independent weight priors do not translate to meaningful behavior on unseen data. As shown by Hafner et al. (2019), standard weight priors fail to constrain functional behavior in OOD regions. A well-behaved, bounded distribution over weights can still produce wildly erratic and overconfident functions just outside the training manifold.
Parameter Space (Weights) Function Space (Outputs)
┌─────────────────────────┐ ┌─────────────────────────┐
│ Symmetric, complex, │ Many-to- │ What we actually care │
│ unintuitive landscapes │ ──One───► │ about: smooth curves, │
│ (e.g., Gaussian prior) │ │ high uncertainty OOD │
└─────────────────────────┘ └─────────────────────────┘
Shifting the Prior to the Output Space
Instead of asking “what should the weights look like?”, function-space regularization asks “what should the predictions look like?”
By defining priors directly over the model's output space, we can explicitly encode desired functional properties (Hafner et al., 2018; Klarner et al., 2023). For example, if a model encounters an input far from the training data, we want its output to revert to a state of maximum uncertainty (e.g., a uniform distribution in classification).
Because this constraint is placed on the outputs, it is entirely architecture-agnostic (Lin et al., 2023). The same functional prior can be applied to a convolutional network, a transformer, or a low-rank adaptation layer (Zhang et al., 2026) without modifying the internal structure of the model.
How It Works: Measurement Sets and Contrastive Objectives
We cannot analytically evaluate a neural network's function across the entire infinite input space. To make function-space regularization tractable, modern methods rely on measurement sets—collections of inputs used specifically to evaluate and guide the model's functional behavior (Lin et al., 2023).
A prominent implementation of this idea is Noise Contrastive Priors (NCPs), introduced by Hafner et al. (2018). The mechanism operates via a simple contrastive objective:
- Train on Clean Data: The model learns the standard input-to-output mapping from the training distribution.
- Fantasize OOD Data: The algorithm generates a measurement set by adding noise to the training inputs, pushing them just outside the data manifold.
- Penalize Overconfidence: The objective function applies a penalty based on the divergence between the model's predictions on this noisy measurement set and a high-uncertainty target distribution (Hafner et al., 2019).
By explicitly teaching the model to express high uncertainty on data points that deviate from the training distribution, the network learns a boundary around the data manifold where its confidence rapidly drops off.
Preserving Calibration under Distribution Shift
When a model encounters a distribution shift, traditional networks remain stubbornly confident. Function-space methods fix this by directly regularizing the predictive distribution.
In Function-Space Variational Inference (FSVI), developed by Rudner et al. (2022), the variational objective optimizes the distribution of outputs directly rather than the distribution of network weights. This ensures that as inputs move further from the training manifold, the predictive entropy naturally expands.
By tying the regularization penalty to specific input locations via Function-Space Empirical Bayes (FS-EB), models achieve near-perfect out-of-distribution detection and significantly lower expected calibration error without the massive computational overhead of full weight-space Bayesian inference (Rudner et al., 2023). Similarly, in simulation-based inference, centering a functional prior on the simulator's parameter prior forces the model to safely revert to a conservative baseline when data is scarce (Delaunoy et al., 2024).
Exploiting Overparameterization for Continual and Multi-Task Learning
Beyond uncertainty estimation, function-space regularization provides a elegant solution to a core challenge in machine learning: how to adapt a model without destroying what it has already learned.
Continual Learning
In sequential task learning, parameter-space regularization (like penalizing changes to weights) causes catastrophic forgetting because it locks parameters in place. However, because neural networks are highly overparameterized, there are many different weight configurations that can produce the exact same output behavior.
Rudner (2022) introduced Sequential Function-Space Variational Inference (S-FSVI) to exploit this property. Instead of penalizing weight changes, S-FSVI regularizes the model's predictions on a measurement set representing previous tasks. This allows the underlying weights to shift significantly to accommodate new tasks, provided they preserve the functional mapping required for past tasks.
Federated and Multi-Task Learning
A similar dynamic occurs when training models on decentralized or non-I.I.D. data. In federated learning, forcing local model parameters to remain close to a global average fails to capture complex classification performance. By regularizing the model's functional outputs instead, the framework ensures consistent predictive behavior across different users while allowing local parameters the flexibility to adapt to user-specific nuances (Li et al., 2022).
Grounding Models with Domain and Physics Priors
Because function-space constraints operate on the same plane as real-world observations, they allow researchers to inject highly interpretable domain knowledge directly into deep learning architectures.
- Physics-Constrained Systems: In robotics and control, Sukhija et al. (2024) utilize low-fidelity physics simulators as functional priors for BNNs. This acts as a powerful regularizer that prevents the model from generating unphysical predictions in unexplored regions of the state space, bridging the sim-to-real gap with fraction of the data.
- Drug Discovery: In molecular property prediction, models often face severe covariate shifts. Klarner et al. (2023) demonstrate that domain-informed functional priors maintain tight calibration when evaluating entirely novel molecular structures, preventing dangerous overconfidence in high-stakes discovery pipelines.
- Epidemic and Trend Forecasting: By comparing current data patterns to a historical reference set in the functional space, frameworks like those proposed by Kamarthi et al. (2021) allow models to flag when an ongoing event deviates from historical analogs, providing transparent, non-parametric uncertainty quantification.
Ultimately, function-space regularization succeeds because it aligns the optimization constraint with the model's actual purpose: generating reliable, well-calibrated predictions across the entire input domain, even when operating at the boundaries of its knowledge (Rothfuss et al., 2021; Acerbi et al., 2024).
Key Foundational Work
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018) — contributed 2 concept elements to this direction.
A Speaker Adaptive DNN Training Approach for Speaker-Independent Acoustic Inversion (2017) — contributed 1 concept elements to this direction.