Analytical Propagation of Uncertainty Through Non-Linear Networks

27 concepts20 papers1 proposal

Deep neural networks excel at mapping deterministic inputs to deterministic outputs. However, in real-world deployments—such as autonomous driving, medical imaging, or speech recognition—inputs are frequently corrupted by noise, and model parameters are inherently uncertain. To build reliable systems, we must propagate entire probability distributions, rather than single data points, through a network's layers.

If a neural network consisted solely of linear transformations, propagating uncertainty would be trivial: passing a Gaussian distribution through a linear layer yields another Gaussian distribution, completely characterized by its transformed mean and covariance. However, deep learning relies fundamentally on non-linear activation functions (such as ReLU, Sigmoid, or Softmax). As A.H. Abdelaziz et al. et al. observe in Uncertainty propagation through deep neural networks, these non-linearities break the chain of analytical tractability, preventing simple input distributions from being easily tracked to the final layer.

The Geometry of Distributional Warping

Why exactly do non-linearities break uncertainty propagation? The core issue is geometric. When a continuous probability distribution passes through a non-linear activation function, its shape is warped into something far more complex.

Consider the Rectified Linear Unit (ReLU), which clips all negative values to zero. If an input distribution spans both positive and negative domains, a ReLU layer collapses the entire negative probability mass into an infinitely dense point mass at zero, while leaving the positive side intact. As Mahyar Fazlyab et al. explain in Probabilistic Verification and Reachability Analysis of Neural Networks via Semidefinite Programming, this transformation warps simple, symmetric input distributions into highly complex, non-Gaussian shapes.

   Gaussian Input             ReLU Activation            Warped Output
       _.._                         |                       |   _.._
     .'    `.                       | /                     | .'    `.
____/        \____   ==========>    |/       ==========>    |/        \____
                                  --+--                   --+-----------
                                                            ^
                                                      (Point mass at 0)

This warping compounds exponentially with network depth. In Input uncertainty propagation through trained neural networks, Paul Monchot et al. demonstrate that non-linear layers transform simple input distributions into intricate, multi-modal output distributions. Furthermore, the authors point out a subtle mathematical hazard: because activations like ReLU map entire regions of the input space to exactly zero, they create regions of zero probability. This causes standard information-theoretic metrics like the Kullback-Leibler (KL) divergence to explode to infinity, frustrating standard distribution-alignment techniques.

This is not a failure of specific approximations, but a fundamental mathematical limitation. In A Complete Characterisation of ReLU-Invariant Distributions, Jan MacDonald et al. prove a definitive "no-go" theorem: no computationally practical, continuous family of probability distributions can remain perfectly invariant under the action of ReLU layers. The actual uncertainty distribution quickly escapes the low-dimensional parameter space of standard distribution families, forcing practitioners to accept lossy approximations.

The Non-Analytical Expectation Hurdle

The geometric warping of distributions creates an immediate roadblock for both training and inference: the inability to compute expectations in closed form. In Bayesian deep learning and robust optimization, we frequently need to integrate over a distribution to compute an expected value—whether calculating the Evidence Lower Bound (ELBO) for variational inference or marginalizing out noise.

This challenge was highlighted early on by L.K. Saul et al. in Mean Field Theory for Sigmoid Belief Networks, where the authors noted that the expectation of non-linear functions (like the log-sigmoid) lacks a closed-form solution even under fully factorized distributions. Without an analytical solution, models cannot directly evaluate their own objective functions.

This "non-analytical expectation hurdle" manifests across several domains:

As Shinji Watanabe et al. summarize in Uncertainty training and decoding methods of deep neural networks based on stochastic representation of enhanced features, these mathematical barriers force a heavy reliance on sampling methods, which trade computational efficiency for directional accuracy.

Tracking Moments and Cross-Correlations

If tracking the entire warped distribution is impossible, a natural alternative is to track only its lowest-order statistical moments: the mean and the covariance. Yet, even this simplified task encounters severe mathematical friction because non-linearities introduce complex interactions between random variables.

In An Analytic Solution to Covariance Propagation in Neural Networks, Oren Wright et al. point out that standard approximations often ignore cross-correlations between hidden units or rely on crude simplifications, resulting in highly inaccurate representations of the output distribution. While Adel Bibi et al. succeeded in deriving exact formulas for the probabilistic moments of piecewise-linear networks in Analytic Expressions for Probabilistic Moments of PL-DNN With Gaussian Input, their approach is tightly bound to Gaussian inputs and piecewise-linear activations.

The moment-propagation problem becomes even more acute when dealing with vector-valued non-linearities, such as the softmax operation in attention mechanisms. As Haoming Jing et al. argue in Kalman Bayesian Transformer, the vector-valued nature of softmax makes exact analytical moment propagation mathematically intractable for exact Bayesian updates, forcing a stark trade-off between theoretical rigor and practical efficiency.

Bypassing Intractability

To circumvent these analytical barriers, researchers have developed three primary paradigms that avoid both raw Monte Carlo sampling and exact integration.

1. Deterministic Parameter Propagation

Instead of sampling weights or activations, we can design networks that natively pass the parameters of a distribution forward through the architecture. For example, in Natural-Parameter Networks: A Class of Probabilistic Neural Networks, Hao Wang 0013 et al. propose modeling weights and neurons as exponential-family distributions. By deterministically propagating their natural parameters layer by layer, the network constructs a lightweight, sampling-free approximation of uncertainty.

2. Convex Relaxations and Semidefinite Programming

If tracking the exact distribution or its moments is impossible, we can instead bound the geometry of the non-linear transformations. In Safety Verification and Robustness Analysis of Neural Networks via Quadratic Constraints and Semidefinite Programming, Mahyar Fazlyab et al. abstract non-linear activation functions into a set of quadratic constraints. This mathematical abstraction transforms the intractable task of reachability analysis—determining the exact set of possible outputs for a given range of inputs—into a solvable convex optimization problem via Semidefinite Programming (SDP), yielding provably tight bounds on network behavior.

3. Distributional Sandwiching

Another bounding strategy focuses on sandwiching the network's output distribution directly. In Exact Upper and Lower Bounds for the Output Distribution of Neural Networks with Random Inputs, Andrey Kofnov et al. address the difficulty of calculating the exact cumulative distribution function (CDF) of a network's output. By bounding a general neural network between two specialized ReLU-based architectures, the authors provide a way to calculate guaranteed upper and lower envelopes for the output distribution under noise, bypassing the need for exact geometric integration.

Key Foundational Work

Ensemble Learning for Multi-Layer Networks (1997) — contributed 1 concept elements to this direction.
Mean Field Theory for Sigmoid Belief Networks (1996) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_non_linear_uncertainty_propagation_fde11cb9,
  title = {non-linear uncertainty propagation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-linear-uncertainty-propagation/rdr_fde11cb9e5304f3fbe26435a67264620},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_non_linear_uncertainty_propagation_fde11cb9_v1,
  title = {non-linear uncertainty propagation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-linear-uncertainty-propagation/rdr_fde11cb9e5304f3fbe26435a67264620/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake