Stochastic Inference Averaging

48 concepts42 papers1 proposal

Deep learning models are traditionally deterministic at test time. Once a network is trained, its weights are frozen, and feeding the same input will always yield the exact same output. However, standard model ensembling—training multiple independent networks and averaging their predictions—consistently outperforms single models by smoothing decision boundaries and reducing variance. Unfortunately, traditional ensembles require $N$ times the computational cost to train and store.

Stochastic Inference Averaging bypasses this bottleneck. Instead of training an explicit ensemble of multiple networks, it introduces stochasticity inside a single network and keeps that randomness active during inference. By executing multiple forward passes on the same input and averaging the results, the model constructs a "virtual ensemble" on the fly.

Deterministic Inference:
Input (x) ───> [ Fixed Network ] ───────────────────────────────> Single Prediction (y)

Stochastic Inference Averaging (Monte Carlo Integration):
               ┌─> [ Network + Noise State 1 ] ──> Prediction 1 ┐
Input (x) ────┼─> [ Network + Noise State 2 ] ──> Prediction 2 ┼─> [ Average ] ──> Robust Prediction (y)
               └─> [ Network + Noise State N ] ──> Prediction N ┘

The Core Intuition: Shaking the Decision Boundary

To understand why this works, think of a neural network's decision boundary as a jagged coastline. In a deterministic network, an input vector is a single point. If that point falls even slightly on the wrong side of a sharp, noisy protrusion—such as an adversarial perturbation or a local artifact—the model will make an error.

When we inject stochasticity into the network's layers, we are effectively shaking that coastline. Each forward pass uses a slightly different internal configuration of the network, meaning the decision boundary shifts slightly every time.

By taking the average of multiple passes, we perform Monte Carlo integration over the model's internal noise distribution. This process mathematically smooths out the high-frequency variations in the decision boundary. The sharp protrusions disappear, leaving a stable, well-calibrated boundary that is far more resilient to noise, out-of-distribution shifts, and adversarial attacks.


Mechanisms of Test-Time Stochasticity

Researchers introduce randomness into models at inference time through several distinct architectural mechanisms, depending on the problem domain.

1. Layer-Level Noise and Active Dropout

Instead of turning off regularization layers at test time, keeping them active creates a dynamic ensemble from a single set of weights.

2. Structural and Permutation Shuffling

Randomizing the processing order or pooling regions forces the network to look at data from diverse structural perspectives.

  • Stochastic Attention Shuffling: Local window Transformers often struggle with global context. To break this barrier, Xiao et al. propose the Random Shuffle Transformer for Image Restoration, which shuffles pixels before applying local attention. At inference time, multiple passes with different random permutations are averaged to stabilize the output and aggregate diverse global perspectives. This mechanism is also leveraged by Xiao et al. in HomoFormer to distribute non-uniform shadow degradation across images.
  • Stochastic Scanning: Jie Xiao applies this concept to state-space models in Non-causal selective state space model for image restoration, performing multiple independent random token scans during testing and averaging the outputs to mitigate the variance of individual permutations.
  • Stochastic Pooling: In computer vision, Lai et al. show in Toward high-performance online HCCR that replacing fixed pooling grids with stochastic pooling regions introduces an internal elastic distortion. Averaging predictions across different random pooling configurations yields the benefits of model ensembling without the cost of training independent networks.

3. Latent Space and Prompt Sampling

Randomness can also be injected at the input, prompt, or parameter level to smooth out prediction spaces.

  • Latent Space Averaging: Bert Huang demonstrates in Structured Output Learning with Conditional Generative Flows that drawing multiple samples from a latent Gaussian distribution and passing them through an inverse flow provides a smoothing effect that avoids optimization artifacts.
  • Stochastic Prompting: In vision-language models, Silin Cheng et al. introduce VaMP, which treats prompts as latent variables sampled from a learned distribution. Averaging multiple stochastic prompt realizations at test time acts as an internal ensemble that enhances robustness against input noise. A similar test-time prompt ensemble strategy is utilized by Xi Xiao et al. in Visual Instance-aware Prompt Tuning.

Why It Works: The Mathematics of Variance and Invariance

Stochastic inference averaging works because it resolves a fundamental tension in stochastic networks: training-time variance vs. test-time stability.

When a network incorporates stochastic components during training—such as the label noise in DisturbLabel by Xie et al., the randomized convolutional layers in Network Randomization by Lee et al., or the stochastic competition layers in STab by Voskou et al.—it learns to be invariant to that noise. The network is effectively trained on an exponential number of noisy versions of the dataset.

However, if you run only a single deterministic pass at test time, you are exposed to the variance of whatever random state that pass happens to land on.

By running multiple passes and averaging them, you compute an empirical expectation:

$$\hat{y} = \frac{1}{M} \sum_{m=1}^{M} f(x; \theta, \epsilon_m)$$

where $\epsilon_m$ represents the random seed or noise state of the $m$-th pass. This suppresses the variance of individual random permutations, allowing the underlying signal to emerge.

This variance reduction is critical across diverse domains:

  • Image Denoising: The Recorrupted-to-Recorrupted (R2R) framework processes multiple recorrupted versions of an input at inference time, aggregating them to enhance the signal-to-noise ratio of the final output.
  • Spatial Localization: Liu et al. show in Video Crowd Localization that Monte Carlo integration mitigates the variance introduced by stochastic sampling in attention mechanisms, yielding stable coordinate outputs.
  • Implicit Generative Ensembles: In Diffusion Models for Implicit Image Segmentation Ensembles, Wolleb et al. leverage the inherent randomness of a diffusion model's reverse sampling path. Running this stochastic process multiple times generates a variety of candidate segmentation masks; averaging them produces a highly accurate final segmentation.

The Trade-off: Latency for Quality

The primary drawback of stochastic inference averaging is computational latency. Running $N$ forward passes takes roughly $N$ times longer than a single pass.

However, this trade-off is highly controllable. Unlike traditional ensembles, which require massive memory overhead to store $N$ distinct sets of model weights, stochastic inference averaging operates on a single set of weights. You can scale the number of inference passes up or down at test time depending on your compute budget, without changing the underlying model. For high-stakes tasks like medical image segmentation or adversarial defense, spending extra compute cycles at inference time to buy a smoother, lower-variance prediction is a highly advantageous trade-off.

Key Foundational Work

Swapout: Learning an ensemble of deep architectures (2016) — contributed 2 concept elements to this direction.
Toward high-performance online HCCR: A CNN approach with DropDistortion, path signature and spatial stochastic max-pooling (2017) — contributed 2 concept elements to this direction.
DisturbLabel: Regularizing CNN on the Loss Layer (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stochastic_inference_averaging_b84664e1,
  title = {stochastic inference averaging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-inference-averaging/rdr_b84664e1a9964840be45c6bb4b31f99f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_stochastic_inference_averaging_b84664e1_v1,
  title = {stochastic inference averaging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-inference-averaging/rdr_b84664e1a9964840be45c6bb4b31f99f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake