TL;DR
The authors introduce CF-Prop, a method that uses characteristic functions to propagate uncertainty through Bayesian ReLU layers exactly and without sampling. By moving to the Fourier domain, they enable CF-VI, a variational inference algorithm that achieves state-of-the-art calibration on CIFAR-100 and accurately captures "tail risk" in heavy-tailed datasets where traditional Gaussian-based approximations fail.
What Problem Does This Paper Address?
Bayesian Neural Networks (BNNs) are the gold standard for quantifying uncertainty, but they are notoriously difficult to scale. Standard training relies on Monte Carlo (MC) sampling, which is computationally expensive and noisy. To solve this, researchers developed "sampling-free" methods like Deterministic Variational Inference (DVI).
However, these fast methods typically rely on a Gaussian assumption: they assume the output of every layer follows a normal distribution. This assumption breaks down in two critical ways:
- Architecture: In layers with a moderate number of neurons (e.g., 50–100), the Central Limit Theorem hasn't "kicked in" yet, leaving the predictive distribution skewed or heavy-tailed.
- Data Regimes: Many real-world problems (like insurance claims or disaster forecasting) involve non-Gaussian likelihoods (Poisson, Gamma) where the most important information lives in the "tails" of the distribution.
As shown in Figure 1, standard methods like DVI and CovProp significantly underestimate these tails, potentially leading to catastrophic failures in safety-critical applications.
Methods
The core innovation is CF-Prop, which represents the distribution of neural network activations using Characteristic Functions (CFs)—the Fourier transform of a probability density function.
- Linear Layers as Products: In the Fourier domain, the sum of independent random variables (the core operation of a linear layer) becomes a simple product of their CFs. This makes propagating uncertainty through weights and biases mathematically elegant.
- The ReLU Challenge: The non-linearity of ReLU is usually the "deal-breaker" for analytical methods. The authors overcome this by using Hilbert transforms to express the action of a ReLU activation in the Fourier domain. This allows them to compute the exact output distribution of a Bayesian ReLU layer without any sampling.
- CF-VI (Variational Inference): The authors show that for a wide class of "exponential-polynomial" likelihoods (including Gaussian, Poisson, and Gamma), the reconstruction cost of the Evidence Lower Bound (ELBO) can be computed in closed form. Instead of expensive integrals or sampling, training involves evaluating the characteristic function at just a few specific points.
Datasets
The authors evaluated CF-VI across several regimes:
- UCI Regression Benchmarks: Standard datasets like Boston Housing, Concrete, and Energy to test general predictive performance.
- CIFAR-10 & CIFAR-100: Image classification tasks using a frozen Wide ResNet-28-10 backbone to test the method as a "Bayesian last layer."
- freMTPL2 (French Motor Insurance): A real-world dataset of insurance claim severities characterized by extremely heavy tails (rare but very expensive claims).
- Synthetic Regression: A toy problem with a "gap" in training data to visualize epistemic uncertainty (Figure 3).
Key Insights and Ideas
- The Gaussian Assumption is Dangerous: The paper demonstrates that even at widths of 128 units, the error in the tails for Gaussian-approximation methods is significant. If you are predicting the probability of a "100-year flood," underestimating the tail by an order of magnitude is a major failure.
- Exactness without Sampling: CF-Prop matches Monte Carlo baselines perfectly (see Figure 2) because it isn't an approximation of the distribution—it is an exact analytical representation.
- Efficiency: Because the ELBO is available in closed form for many likelihoods, CF-VI adds only a modest overhead (about 1.8x) compared to a deterministic network, while being significantly faster than Monte Carlo methods during both training and inference.
Key Results
- CIFAR-100 Calibration: CF-VI achieved the lowest Expected Calibration Error (ECE) of all tested methods (3.1%), a 35% improvement over the next best method (Bayes by Backprop).
- Insurance Tail Risk: On the Gamma regression task (freMTPL2), CF-VI was the only method to accurately capture tail uncertainty. For high-risk individuals, other sampling-free methods underestimated the probability of a large claim by a factor of $10^8$.
- Out-of-Distribution Robustness: In synthetic tests with missing data (Figure 3), CF-VI correctly increased its uncertainty in the "gap" region, whereas DVI and CovProp remained overconfident.
Limitations and Future Work
- Single-Layer Focus: Currently, the exact propagation is derived for a single Bayesian layer (often used as a "Bayesian head" or last layer). While it can be used in deeper networks, the "exactness" doesn't yet propagate through multiple non-linear layers without compounding assumptions.
- Mean-Field Assumption: The current derivation assumes that activations are independent (mean-field). It does not yet capture correlations between neurons, which the authors identify as a primary area for future work.
- Scaling to Many Layers: The authors suggest that other integral transforms, like the Mellin transform, might be combined with their approach to describe the full forward pass of deep networks in the future.
Summary updated