Analytic Uncertainty Propagation in Deep Neural Networks

This direction is split into other directions: non-linear uncertainty propagation. Its last version stays readable and citable here.

Version 1 of 140 papers43 conceptsfrozen 2026-10-04

Standard deep neural networks are "point-estimate" machines: they take a single input vector and produce a single output prediction. However, in high-stakes domains like autonomous driving or medical diagnostics, knowing the model's confidence is as important as the prediction itself. While Bayesian Neural Networks (BNNs) offer a framework for this, they typically rely on computationally expensive Monte Carlo sampling—running the same input through the network hundreds of times with different weights to see how the output varies.

A growing body of research seeks to bypass this "sampling tax" by treating uncertainty as a signal that can be propagated deterministically through the network's layers, much like the activations themselves.

The Non-Linearity Wall

The core challenge of uncertainty propagation is the mathematical transformation of probability distributions. As A.H. Abdelaziz and Shinji Watanabe observed, while linear layers (like convolutions or fully connected layers) transform Gaussian inputs into Gaussian outputs predictably, non-linear activation functions like ReLU, Sigmoid, or tanh break this symmetry.

When a Gaussian distribution passes through a ReLU, it is "clipped" at zero, resulting in a rectified Gaussian distribution that is no longer symmetric. As these distributions pass through subsequent layers, they become increasingly complex and multi-modal. Jan MacDonald and Stephan Wäldchen provided a definitive "no-go" theorem for this problem, proving that no computationally practical family of distributions is perfectly invariant under ReLU transformations. This means that exact uncertainty propagation is, in the general case, analytically intractable.

Deterministic Moment Matching

To overcome this intractability, researchers use Assumed Density Filtering (ADF) or moment matching. Instead of trying to track the exact shape of the distribution, the network tracks only its summary statistics—usually the mean and variance (the first and second "moments").

At each layer, the network calculates how the mean and variance of the input distribution should change. For linear operations, this is straightforward. For non-linearities, researchers derive analytical approximations:

  • Adel Bibi and Bernard Ghanem derived exact formulas for the moments of shallow ReLU networks, providing a "flashlight" into how noise propagates.
  • Oren Wright and Yorie Nakahira recently extended this to provide exact analytic solutions for covariance propagation, allowing models to track how different features become correlated as they move through the network.
  • Manan Gandhi developed specific methods for the tanh function, which is critical for recurrent architectures like Echo State Networks.

Sampling-Free Architectures

Beyond approximating standard networks, new architectures are designed from the ground up to be "uncertainty-aware."

Natural-Parameter Networks (NPN), introduced by Hao Wang, treat both weights and neurons as distributions from the exponential family. Instead of propagating values, they propagate the "natural parameters" of these distributions. Similarly, Seong Jae Hwang and Melih Kandemir developed SP-GRU, a probabilistic Gated Recurrent Unit that achieves a 10x speedup over sampling-based methods by deterministically propagating distribution parameters through the gates.

In the context of auto-encoders, Minmin Chen and Kilian Q Weinberger developed Marginalized Denoising Auto-encoders (mDAE). Rather than manually adding noise to data (corrupting it) and asking the model to fix it, they mathematically "marginalize out" the noise. This effectively trains the model on an infinite number of noisy variations simultaneously, leading to massive speedups in training time.

Why It Works: The Intuition of Sigma Points

The intuition behind many of these deterministic methods is similar to the Unscented Transform (UT), explored in automotive contexts by Abraham Elias Ortega Paredes.

If you have a distribution and a non-linear function, you don't need to sample every possible point. Instead, you can pick a small, representative set of "sigma points" that capture the mean and spread of the distribution. You pass only these points through the non-linearity and then reconstruct the new distribution from where they land. Analytic propagation takes this a step further by finding the mathematical limit of this process, allowing the network to "know" how the spread of its predictions changes without ever having to actually roll the dice.

Applications in Robustness and Safety

The ability to track uncertainty analytically has profound implications for model reliability:

  1. Adversarial Defense: By understanding how input noise translates to output variance, researchers like Adel Bibi can craft more sophisticated attacks and, conversely, Steven Adams can provide formal certifications of robustness.
  2. Efficiency: Methods like Variance Backpropagation (VBP) by Melih Kandemir allow for training Bayesian models with the stability and speed of standard networks.
  3. Safety: In autonomous systems, Matthew Wicker and Luca Laurenti use these propagated distributions to provide "reach-avoid" guarantees, ensuring a system stays within safe bounds even when its sensors are noisy.

By moving from stochastic sampling to analytic propagation, the field is turning uncertainty from a computational burden into a tractable, first-class citizen of the neural network forward pass.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_analytic_uncertainty_propagation_in_deep_1e72ab10,
  title = {Analytic Uncertainty Propagation in Deep Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/analytic-uncertainty-propagation-in-deep-neural-networks/rdr_1e72ab104c94475fb202985bdeadc3aa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_analytic_uncertainty_propagation_in_deep_1e72ab10_v1,
  title = {Analytic Uncertainty Propagation in Deep Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/analytic-uncertainty-propagation-in-deep-neural-networks/rdr_1e72ab104c94475fb202985bdeadc3aa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}