Deterministic Moment Propagation for Fast Probabilistic Deep Learning

32 concepts26 papers1 proposal

Standard deep neural networks are deterministic point-estimators: they take a single input and produce a single output. To capture uncertainty—essential for safety-critical applications or active learning—researchers typically turn to Bayesian Neural Networks (BNNs). However, BDL usually relies on Monte Carlo sampling, which requires passing data through the network dozens or hundreds of times to build a distribution.

A growing body of research proposes a more efficient alternative: Deterministic Moment Propagation. Instead of sampling, these methods treat every activation as a random variable and propagate its statistical moments (typically the mean and variance) through the network in a single forward pass.

The Central Limit Theorem as a Foundation

The core intuition behind this approach is that while the input to a network might be complex, the aggregate input to any single neuron often simplifies. As Barber et al. observed, the total weighted input to a node in a layered network can be modeled as a Gaussian distribution by leveraging the Central Limit Theorem (CLT). Because a neuron sums many independent (or weakly dependent) signals from the previous layer, the distribution of that sum tends toward a bell curve.

This approximation is remarkably robust. Wu et al. demonstrated that the Gaussian approximation for hidden layer activations remains highly accurate in networks with as few as 25 units per layer. By assuming pre-activations are Gaussian, researchers like Soudry et al. developed "parameter-free" training algorithms that propagate means and variances to calculate expected layer outputs analytically, eliminating the need for traditional backpropagation in some contexts.

Assumed Density Filtering and Moment Matching

The primary challenge to this framework is the non-linear activation function (e.g., ReLU, Sigmoid). When a Gaussian distribution passes through a non-linearity, the resulting output is no longer Gaussian. To maintain a tractable "closed-loop" system, the network must project this complex distribution back into a simpler family—a process known as Assumed Density Filtering (ADF).

As described by Gast et al., ADF treats every activation as a distribution and propagates them using closed-form expressions for mean and variance. When the distribution becomes non-Gaussian after an activation, the system performs moment matching: it calculates the exact mean and variance of the non-linear output and then "assumes" a new Gaussian with those same statistics for the next layer. Finzi et al. and Akgül et al. utilize this technique to ensure that uncertainty information flows through deep architectures without the exponential complexity of tracking arbitrary probability shapes.

Analytic Solutions for Non-Linearities

To make this propagation deterministic and fast, researchers have derived exact mathematical "flashlights" for common layers:

  • Piecewise Linear Layers: Bibi et al. derived exact formulas for the first and second moments of outputs in piecewise linear DNNs under Gaussian noise.
  • ReLU and Max-Pooling: Likowski et al. and Shekhovtsov et al. provided stable, closed-form analytical formulas for the expected value of ReLUs and max-related operations, allowing for probabilistic reasoning without Monte Carlo overhead.
  • Covariance Tracking: While many methods only track variance (the diagonal of the covariance matrix), Wright et al. introduced an analytic solution for full covariance propagation using Taylor series expansions, enabling a more precise understanding of how neurons correlate.

Why It Works: Speed and Calibration

The shift from stochastic sampling to analytical integration offers three major advantages:

  1. Efficiency: By replacing hundreds of forward passes with a single pass that propagates two values ($\mu, \sigma^2$) instead of one, training and inference become significantly faster. Wang et al. showed that a deterministic approximation of Dropout could achieve a tenfold increase in training speed while maintaining accuracy.
  2. Calibration: Because the network explicitly tracks uncertainty, it "knows when it doesn't know." Thiagarajan et al. showed that propagating input uncertainties allows models to decompose prediction variance back to specific features, improving interpretability.
  3. Stability: Deterministic propagation removes the "noise" inherent in sampling-based gradients. Wu et al. found that this leads to more robust Bayesian Neural Networks that are easier to initialize and train.

Recent extensions of this idea have moved beyond simple MLPs to complex architectures like Transformers Jing et al. and Implicit Neural Representations Chen et al., suggesting that deterministic moment propagation is a general-purpose tool for bringing rigorous uncertainty quantification to the entire deep learning stack.

Key Foundational Work

Gaussian Fields for Approximate Inference in Layered Sigmoid Belief Networks (1999) — contributed 2 concept elements to this direction.
Propagation of uncertainty through multilayer perceptrons for robust automatic speech recognition (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gaussian_activation_approximation_d4b797a3,
  title = {Gaussian activation approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gaussian-activation-approximation/rdr_d4b797a358e344559c88a801a90787c7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_gaussian_activation_approximation_d4b797a3_v1,
  title = {Gaussian activation approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gaussian-activation-approximation/rdr_d4b797a358e344559c88a801a90787c7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake