In standard deep learning, we treat activations as fixed scalar values. When we want to account for uncertainty—whether from noisy inputs or uncertain model weights—the traditional approach is Monte Carlo (MC) sampling: we run the network dozens or hundreds of times with different noise samples and average the results. While effective, this is computationally expensive and introduces gradient noise that can destabilize training.
A growing body of research proposes a more efficient alternative: Deterministic Moment Propagation. Instead of passing point estimates through the network, these models propagate the parameters of a probability distribution (typically the mean and variance) through every layer. By deriving closed-form expressions for how these "moments" change, we can quantify uncertainty in a single forward pass without ever drawing a sample.
From Points to Distributions
The core intuition behind this direction is to replace the standard activation $h = \sigma(Wx + b)$ with a probabilistic version. If the input $x$ is a random variable with mean $\mu_x$ and variance $\sigma^2_x$, the goal is to find the mean and variance of the output $h$ after it passes through the weights and the non-linear activation function $\sigma$.
For linear layers, this is mathematically straightforward. As noted by Justin Bayer and Patrick van der Smagt in their work on Fast Adaptive Weight Noise (FAWN), the Central Limit Theorem often allows us to assume that the pre-activations (the sum of many weighted inputs) follow a Gaussian distribution. This simplifies the problem: we only need to track how the first two moments—the mean and the variance—evolve.
Handling the Non-Linearity Bottleneck
The real challenge arises at the non-linear activation functions (like ReLU or Sigmoid). When a Gaussian distribution passes through a non-linear "hinge" like ReLU, the output is no longer Gaussian; it becomes a truncated or skewed distribution.
To keep the computation tractable across many layers, researchers use a technique called Assumed Density Filtering (ADF) or Moment Matching. As explained by Jochen Gast and Stefan Roth in Lightweight Probabilistic Deep Networks, we project the complex, non-Gaussian output back onto a simpler family (usually a Gaussian) that shares the same mean and variance.
Different researchers have derived specific solutions for different activations:
- ReLU: José Miguel Hernández-Lobato in Probabilistic Backpropagation and Christos Tzelepis in his Technical Report show that for ReLU, the output mean and variance can be calculated exactly using the Gaussian error function ($\text{erf}$).
- Piecewise Linear Functions: Adel Bibi and Bernard Ghanem derived exact analytic expressions for the moments of any piecewise linear network.
- Tanh and Sigmoid: Alexander Shekhovtsov and Boris Flach developed numerical approximations for squashing functions, while Manan Gandhi used piecewise cubic polynomials to propagate uncertainty through tanh in recurrent networks.
Why It Works: Stability and Speed
The primary advantage of this approach is the elimination of sampling noise. In traditional Bayesian Neural Networks (BNNs), the gradients used for training are stochastic because they depend on random samples. This makes training "finicky" and sensitive to hyperparameters.
By using deterministic propagation, Anqi Wu and Richard E. Turner demonstrate in Deterministic Variational Inference that we can achieve much smoother convergence. Because the entire forward pass is a differentiable, analytic function of the input statistics, the gradients are exact rather than estimated.
Furthermore, the speed gains are massive. Seong Jae Hwang and Vikas Singh reported a 10x speedup in Gated Recurrent Units (GRUs) by replacing sampling with moment matching. More recently, Rui Li and Arno Solin have extended these techniques to Vision Transformers and GPT-2, showing that local linearization allows for "streamlined" Bayesian prediction in architectures previously considered too complex for analytical treatment.
Applications Beyond Uncertainty
While the initial motivation was Bayesian inference, deterministic moment propagation has found diverse applications:
- Adversarial Robustness: By modeling input perturbations as Gaussian noise and propagating that variance, researchers like Aysegul Dundar have built CNNs that are inherently more resilient to adversarial attacks.
- Missing Data: Marcin Przewiezlikowski and colleagues developed MisConv, which treats missing pixels as probability distributions and calculates the "expected" convolution, allowing networks to process incomplete images without explicit imputation.
- Reinforcement Learning: In offline RL, Melih Kandemir and Manuel Haußmann use moment matching in MOMBO to track how uncertainty compounds over long trajectories, leading to more stable policy optimization.
- Interpretability: The Rate-Distortion Explanation (RDE) framework uses ADF to analytically propagate the effects of masking different input features, identifying which parts of an image are most "relevant" to a model's decision.
By treating neural networks as processors of distributions rather than processors of points, this research direction provides a mathematically rigorous path toward models that are not only faster but also "know what they don't know."