Deterministic Moment Propagation for Sampling-Free Uncertainty

This direction is split into other directions: Gaussian activation approximation. Its last version stays readable and citable here.

Version 1 of 160 papers76 conceptsfrozen 2026-10-04

In standard deep learning, we treat activations as fixed scalar values. When we want to account for uncertainty—whether from noisy inputs or uncertain model weights—the traditional approach is Monte Carlo (MC) sampling: we run the network dozens or hundreds of times with different noise samples and average the results. While effective, this is computationally expensive and introduces gradient noise that can destabilize training.

A growing body of research proposes a more efficient alternative: Deterministic Moment Propagation. Instead of passing point estimates through the network, these models propagate the parameters of a probability distribution (typically the mean and variance) through every layer. By deriving closed-form expressions for how these "moments" change, we can quantify uncertainty in a single forward pass without ever drawing a sample.

From Points to Distributions

The core intuition behind this direction is to replace the standard activation $h = \sigma(Wx + b)$ with a probabilistic version. If the input $x$ is a random variable with mean $\mu_x$ and variance $\sigma^2_x$, the goal is to find the mean and variance of the output $h$ after it passes through the weights and the non-linear activation function $\sigma$.

For linear layers, this is mathematically straightforward. As noted by Justin Bayer and Patrick van der Smagt in their work on Fast Adaptive Weight Noise (FAWN), the Central Limit Theorem often allows us to assume that the pre-activations (the sum of many weighted inputs) follow a Gaussian distribution. This simplifies the problem: we only need to track how the first two moments—the mean and the variance—evolve.

Handling the Non-Linearity Bottleneck

The real challenge arises at the non-linear activation functions (like ReLU or Sigmoid). When a Gaussian distribution passes through a non-linear "hinge" like ReLU, the output is no longer Gaussian; it becomes a truncated or skewed distribution.

To keep the computation tractable across many layers, researchers use a technique called Assumed Density Filtering (ADF) or Moment Matching. As explained by Jochen Gast and Stefan Roth in Lightweight Probabilistic Deep Networks, we project the complex, non-Gaussian output back onto a simpler family (usually a Gaussian) that shares the same mean and variance.

Different researchers have derived specific solutions for different activations:

Why It Works: Stability and Speed

The primary advantage of this approach is the elimination of sampling noise. In traditional Bayesian Neural Networks (BNNs), the gradients used for training are stochastic because they depend on random samples. This makes training "finicky" and sensitive to hyperparameters.

By using deterministic propagation, Anqi Wu and Richard E. Turner demonstrate in Deterministic Variational Inference that we can achieve much smoother convergence. Because the entire forward pass is a differentiable, analytic function of the input statistics, the gradients are exact rather than estimated.

Furthermore, the speed gains are massive. Seong Jae Hwang and Vikas Singh reported a 10x speedup in Gated Recurrent Units (GRUs) by replacing sampling with moment matching. More recently, Rui Li and Arno Solin have extended these techniques to Vision Transformers and GPT-2, showing that local linearization allows for "streamlined" Bayesian prediction in architectures previously considered too complex for analytical treatment.

Applications Beyond Uncertainty

While the initial motivation was Bayesian inference, deterministic moment propagation has found diverse applications:

  1. Adversarial Robustness: By modeling input perturbations as Gaussian noise and propagating that variance, researchers like Aysegul Dundar have built CNNs that are inherently more resilient to adversarial attacks.
  2. Missing Data: Marcin Przewiezlikowski and colleagues developed MisConv, which treats missing pixels as probability distributions and calculates the "expected" convolution, allowing networks to process incomplete images without explicit imputation.
  3. Reinforcement Learning: In offline RL, Melih Kandemir and Manuel Haußmann use moment matching in MOMBO to track how uncertainty compounds over long trajectories, leading to more stable policy optimization.
  4. Interpretability: The Rate-Distortion Explanation (RDE) framework uses ADF to analytically propagate the effects of masking different input features, identifying which parts of an image are most "relevant" to a model's decision.

By treating neural networks as processors of distributions rather than processors of points, this research direction provides a mathematically rigorous path toward models that are not only faster but also "know what they don't know."

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_deterministic_moment_propagation_for_sam_87ef500d,
  title = {Deterministic Moment Propagation for Sampling-Free Uncertainty},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deterministic-moment-propagation-for-sampling-free-uncertainty/rdr_87ef500d3c0d485895728b6df6eaa3cb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_deterministic_moment_propagation_for_sam_87ef500d_v1,
  title = {Deterministic Moment Propagation for Sampling-Free Uncertainty},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deterministic-moment-propagation-for-sampling-free-uncertainty/rdr_87ef500d3c0d485895728b6df6eaa3cb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}