Exact Bayesian layers via characteristic functions

William H. W. Thompson, Guillaume St-Onge, Laurent Hébert-Dufresne, Jean-Gabriel YoungarXiv 2026 · arXiv

TL;DR

The authors introduce CF-Prop, a method that uses characteristic functions to propagate uncertainty through Bayesian ReLU layers exactly and without sampling. By moving to the Fourier domain, they enable CF-VI, a variational inference algorithm that achieves state-of-the-art calibration on CIFAR-100 and accurately captures "tail risk" in heavy-tailed datasets where traditional Gaussian-based approximations fail.

What Problem Does This Paper Address?

Bayesian Neural Networks (BNNs) are the gold standard for quantifying uncertainty, but they are notoriously difficult to scale. Standard training relies on Monte Carlo (MC) sampling, which is computationally expensive and noisy. To solve this, researchers developed "sampling-free" methods like Deterministic Variational Inference (DVI).

However, these fast methods typically rely on a Gaussian assumption: they assume the output of every layer follows a normal distribution. This assumption breaks down in two critical ways:

  1. Architecture: In layers with a moderate number of neurons (e.g., 50–100), the Central Limit Theorem hasn't "kicked in" yet, leaving the predictive distribution skewed or heavy-tailed.
  2. Data Regimes: Many real-world problems (like insurance claims or disaster forecasting) involve non-Gaussian likelihoods (Poisson, Gamma) where the most important information lives in the "tails" of the distribution.

As shown in Figure 1, standard methods like DVI and CovProp significantly underestimate these tails, potentially leading to catastrophic failures in safety-critical applications.

Figure 1: Predictive tails are non-Gaussian for moderate-width networks. A network with two hidden layers of h units each (a deterministic layer followed by a Bayesian layer) is trained on UCI Energy with a 90/10 held-out split, using CF-VI with a mean-field Gaussian variational family and prior and a Gaussian likelihood. 20 networks are trained per width. Top: z-scored predictive density of CF-Prop (purple), DVI (teal, solid), CovProp (teal, dashed), and Monte Carlo (grey) with layers of (a) 8, (b) 64, and (c) 256 units. The insets compare the tails up to 8σ from the mean. Bottom: Total variation (TV) distance between the sampling-free approximation and Monte Carlo predictions, showing (d) the mean (filled) and worst-case (maximum, open) TV over test points.

Methods

The core innovation is CF-Prop, which represents the distribution of neural network activations using Characteristic Functions (CFs)—the Fourier transform of a probability density function.

  • Linear Layers as Products: In the Fourier domain, the sum of independent random variables (the core operation of a linear layer) becomes a simple product of their CFs. This makes propagating uncertainty through weights and biases mathematically elegant.
  • The ReLU Challenge: The non-linearity of ReLU is usually the "deal-breaker" for analytical methods. The authors overcome this by using Hilbert transforms to express the action of a ReLU activation in the Fourier domain. This allows them to compute the exact output distribution of a Bayesian ReLU layer without any sampling.
  • CF-VI (Variational Inference): The authors show that for a wide class of "exponential-polynomial" likelihoods (including Gaussian, Poisson, and Gamma), the reconstruction cost of the Evidence Lower Bound (ELBO) can be computed in closed form. Instead of expensive integrals or sampling, training involves evaluating the characteristic function at just a few specific points.

Datasets

The authors evaluated CF-VI across several regimes:

  • UCI Regression Benchmarks: Standard datasets like Boston Housing, Concrete, and Energy to test general predictive performance.
  • CIFAR-10 & CIFAR-100: Image classification tasks using a frozen Wide ResNet-28-10 backbone to test the method as a "Bayesian last layer."
  • freMTPL2 (French Motor Insurance): A real-world dataset of insurance claim severities characterized by extremely heavy tails (rare but very expensive claims).
  • Synthetic Regression: A toy problem with a "gap" in training data to visualize epistemic uncertainty (Figure 3).

Key Insights and Ideas

  • The Gaussian Assumption is Dangerous: The paper demonstrates that even at widths of 128 units, the error in the tails for Gaussian-approximation methods is significant. If you are predicting the probability of a "100-year flood," underestimating the tail by an order of magnitude is a major failure.
  • Exactness without Sampling: CF-Prop matches Monte Carlo baselines perfectly (see Figure 2) because it isn't an approximation of the distribution—it is an exact analytical representation.
  • Efficiency: Because the ELBO is available in closed form for many likelihoods, CF-VI adds only a modest overhead (about 1.8x) compared to a deterministic network, while being significantly faster than Monte Carlo methods during both training and inference.
Figure 2: CF-Prop (purple) produces predictive densities that perfectly overlap with Monte Carlo samples (grey) across various weight distributions, including skewed and multimodal ones.

Key Results

  • CIFAR-100 Calibration: CF-VI achieved the lowest Expected Calibration Error (ECE) of all tested methods (3.1%), a 35% improvement over the next best method (Bayes by Backprop).
  • Insurance Tail Risk: On the Gamma regression task (freMTPL2), CF-VI was the only method to accurately capture tail uncertainty. For high-risk individuals, other sampling-free methods underestimated the probability of a large claim by a factor of $10^8$.
  • Out-of-Distribution Robustness: In synthetic tests with missing data (Figure 3), CF-VI correctly increased its uncertainty in the "gap" region, whereas DVI and CovProp remained overconfident.
Figure 3: In a regression task with a data gap (grey region), CF-VI (purple) matches the high uncertainty of Monte Carlo samples, while DVI and CovProp (teal) severely underestimate the variance.

Limitations and Future Work

  • Single-Layer Focus: Currently, the exact propagation is derived for a single Bayesian layer (often used as a "Bayesian head" or last layer). While it can be used in deeper networks, the "exactness" doesn't yet propagate through multiple non-linear layers without compounding assumptions.
  • Mean-Field Assumption: The current derivation assumes that activations are independent (mean-field). It does not yet capture correlations between neurons, which the authors identify as a primary area for future work.
  • Scaling to Many Layers: The authors suggest that other integral transforms, like the Mellin transform, might be combined with their approach to describe the full forward pass of deep networks in the future.

Summary updated

Abstract

Bayesian neural networks provide principled uncertainty estimates, but training them with Monte Carlo sampling can be expensive. Sampling-free variational inference algorithms make training Bayesian neural networks feasible, but existing approaches assume a Gaussian output distribution, which can be inaccurate in applied problems. Here, we introduce CF-Prop, an uncertainty propagation method that uses characteristic functions to compute the exact output distribution of a Bayesian ReLU layer, without any sampling, even in non-Gaussian settings. Because CF-Prop propagates characteristic functions, the evidence lower bound---the objective of variational training---can be computed in closed form for a broad class of likelihoods, including the Gaussian, Poisson, and Gamma. This allows to implement an efficient resulting variational inference algorithm, CF-VI. On CIFAR-100, CF-VI achieves the lowest calibration error of any tested method, improving on the next best method by about 35%. On a heavy-tailed insurance claim dataset, CF-VI accurately captures tail uncertainty that is severely underestimated by both other sampling-free methods and Monte Carlo methods.
BibTeX
@misc{thompson2026exact,
  author = {William H. W. Thompson and Guillaume St-Onge and Laurent Hébert-Dufresne and Jean-Gabriel Young},
  title = {{Exact Bayesian layers via characteristic functions}},
  year = {2026},
  eprint = {2611.26426},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26426},
}
Figure