Deep neural networks owe much of their real-world success to the efficiency of the deterministic forward pass. Modern hardware is highly optimized to take an input, propagate it through a sequence of layers, and produce a prediction in milliseconds. However, this single-pass paradigm possesses a fundamental vulnerability: it is brittle. A single forward pass offers no mathematical guarantees against adversarial manipulations, provides poor uncertainty calibration, and can easily be deceived by out-of-distribution inputs or hidden backdoors.
To overcome these limitations, researchers have increasingly turned to multi-pass inference. Instead of relying on a single deterministic prediction, multi-pass frameworks query a model dozens, hundreds, or thousands of times using variations of the input—such as random perturbations, structural augmentations, or sequential reasoning steps—and aggregate the results to form a consensus.
While multi-pass inference provides unprecedented reliability, it introduces a severe latency bottleneck. Because standard deep learning architectures are designed for single-pass execution, evaluating numerous variants scales the computational cost and inference time linearly (or worse), creating a substantial barrier for real-time deployment.
The Mechanics of Randomized Smoothing and the Sampling Tax
To understand why multi-pass inference is so computationally expensive, it is best to examine its most mathematically rigorous manifestation: randomized smoothing. Pioneered by Cohen et al. in Certified Adversarial Robustness via Randomized Smoothing, randomized smoothing converts any standard classifier into a certifiably robust one by evaluating it over a distribution of inputs perturbed by random noise.
The intuition behind this mechanism is elegant. Adversarial attacks typically exploit highly localized, sharp distortions in a model's decision boundary. By adding noise (such as isotropic Gaussian noise) to an input and taking a majority vote over the resulting predictions, the defense effectively "smooths" out these sharp adversarial edges, ensuring the model's output remains constant within a provable geometric radius.
Standard Inference (Single Pass):
Input ──> [ Model ] ──> Brittle Prediction (Vulnerable to sharp boundary shifts)
Multi-Pass Inference / Randomized Smoothing:
┌─> Input + Noise_1 ─> [ Model ] ─┐
Input ───┼─> Input + Noise_2 ─> [ Model ] ─┼─> [ Majority Vote / Aggregation ] ──> Robust Certified Output
└─> Input + Noise_N ─> [ Model ] ─┘
However, the price of this mathematical guarantee is a massive sampling tax. As Jamie Hayes demonstrates in Extensions and limitations of randomized smoothing for robustness guarantees, accurately estimating the model's output distribution under noise requires a high number of Monte Carlo samples. To achieve tight certified bounds with high statistical confidence, defenses must evaluate thousands of noisy variations per input.
Bai Li 0001 et al. echo this challenge in Certified Adversarial Robustness with Additive Noise, noting that estimating class probabilities under stochastic noise demands highly redundant forward passes, which contrasts sharply with the efficiency of standard single-pass neural network architectures. Even when avoiding model retraining by prepending a denoiser to a pretrained classifier—as shown by Hadi Salman et al. in Denoised Smoothing: A Provable Defense for Pretrained Classifiers—the core sampling overhead remains a fundamental barrier to practical deployment.
The Proliferation of Multi-Pass Latency Across Domains
This latency bottleneck is not unique to adversarial computer vision; it is a recurring tax across diverse machine learning paradigms whenever safety, calibration, or reasoning is required.
1. Uncertainty Quantification and Calibration
Deep networks are notoriously overconfident when encountering data that differs from their training set. To mitigate this, Test-Time Augmentation (TTA) evaluates multiple augmented versions of an input to quantify uncertainty. Achim Hekler et al. demonstrate in Test Time Augmentation Meets Post-hoc Calibration: Uncertainty Quantification under Real-World Conditions that while TTA significantly improves uncertainty estimates, generating and processing these multiple versions leads to a linear increase in computational cost and latency. Similarly, the post-hoc calibration method developed by Lei Wang et al. in Optimizing Calibration by Gaining Aware of Prediction Correctness relies on multiple input transformations to sense prediction correctness at test time, multiplying the required forward passes relative to standard single-pass scaling techniques.
2. Large Language Models and Prompting
In the natural language domain, multi-pass inference is frequently used to defend against adversarial prompts or to elicit structured reasoning. Alexander Robey adapts randomized smoothing to text in SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks, showing that randomly perturbing input prompts and aggregating responses via a majority vote effectively neutralizes jailbreaks. However, this multi-sample defense strategy incurs significant computational overhead and latency by requiring the model to process numerous variations of a single input.
Beyond safety, multi-pass prompting for reasoning or translation—such as the self-translation mechanism analyzed by Julen Etxaniz et al. in Do Multilingual Language Models Think Better in English? or the intermediate scene-graph generation proposed by Chancharik Mitra in Compositional Chain-of-Thought Prompting for Large Multimodal Models—inherently doubles or multiplies token expenditure and inference time, complicating high-throughput deployment.
3. Backdoor and Out-of-Distribution Detection
Detecting hidden triggers or out-of-domain inputs frequently relies on multi-pass verification. For instance, Ayush Gupta et al. develop Polysemantic Dropout: Conformal OOD Detection for Specialized LLMs, which detects out-of-domain queries via iterative neuron deactivation across multiple forward passes. In graph domains, Xiao Yang et al. introduce GraphProt: Certified Black-Box Shielding Against Backdoored Graph Models, which relies on multiple subgraph queries during inference to deactivate malicious triggers. In both cases, generating a statistical profile for verification through repeated model queries introduces a substantial latency penalty compared to standard single-pass processing.
Why Parallelism Isn't a Complete Cure
A common counterargument is that multi-pass inferences are independent and can therefore be executed in parallel. While parallel execution can mitigate raw latency if unlimited hardware is available, it does not eliminate the underlying resource constraints.
Running hundreds of forward passes simultaneously requires massive hardware concurrency, multiplying memory consumption, energy costs, and hardware provisioning requirements. For edge devices, autonomous vehicles, or high-throughput production APIs, this resource inflation is often just as disqualifying as sequential latency. The fundamental challenge remains: multi-pass inference trades raw computational efficiency for statistical confidence and security.
Emerging Strategies to Mitigate the Multi-Pass Tax
To make these robust frameworks practical, researchers are actively exploring ways to break the linear relationship between statistical confidence and inference passes.
One frontier involves shifting aggregation from the input space to the representation space. For example, Emanuele Ballarin et al. propose the CARSO framework in Blending adversarial training and representation-conditional purification via aggregation improves adversarial robustness, which aggregates multiple image reconstructions via a specialized logit product, aiming to optimize the efficiency of multi-pass aggregation.
Another radical departure leverages alternative computing paradigms. Tom Wollschläger et al. demonstrate in Discrete Randomized Smoothing Meets Quantum Computing that by using Quantum Amplitude Estimation (QAE), it is possible to achieve a quadratic reduction in the number of model evaluations required for discrete randomized smoothing. By changing how we compute statistical confidence, these approaches aim to preserve the rigorous guarantees of multi-pass inference without paying its prohibitive computational price.