Detecting Malicious Inputs via Predictive Uncertainty

39 concepts32 papers2 proposals

Deep neural networks are notoriously overconfident. A standard deterministic model maps an input to a single point in its logit space, often assigning a 99% probability to an entirely incorrect class when presented with an adversarial perturbation or an out-of-distribution anomaly. This blind spot stems from a lack of internal calibration: the model knows what it thinks, but not how sure it is.

To resolve this, a powerful paradigm has emerged: using predictive uncertainty as a diagnostic runtime signal. By measuring metrics like Shannon entropy, predictive variance, or mutual information across model variations or input perturbations, we can expose the hidden structural anomalies of malicious inputs. Crucially, this approach yields two distinct statistical signatures depending on the threat model: a massive spike in uncertainty for adversarial and anomalous inputs, or an unnaturally rigid lack of uncertainty for backdoored inputs.

The Adversarial Signature: Spiking Uncertainty

Why do adversarial examples fool standard networks? Because they exploit the brittle, high-dimensional decision boundaries of deterministic models. However, if we look at the model through a probabilistic lens—such as a Bayesian Neural Network (BNN) or an ensemble—these manipulated inputs reveal themselves.

When an input is clean and familiar, different plausible parameter configurations (or different models in an ensemble) broadly agree on the classification. The predictive variance is low. But when an adversarial example or an out-of-distribution sample is introduced, it typically lands in an under-sampled, highly ambiguous region of the feature space.

As shown by Yingzhen Li et al. in Dropout Inference in Bayesian Neural Networks with Alpha-divergences, BNNs optimized with mass-covering objectives exhibit a sharp spike in predictive entropy when exposed to adversarial perturbations. This transforms a confident misclassification into a clear signal of high uncertainty. Similarly, Christos Louizos et al. leveraged Multiplicative Normalizing Flows in Multiplicative Normalizing Flows for Variational Bayesian Neural Networks to capture complex, multi-modal posterior distributions, demonstrating that rising input perturbations naturally trigger a corresponding rise in predictive entropy.

This phenomenon extends across diverse modalities and tasks:

The Ensemble Disagreement Mechanism

Instead of relying on a single Bayesian model, we can use an ensemble of diverse models to capture this uncertainty. When an input is out-of-distribution, the individual models diverge in their predictions, creating a high-disagreement state.

In Robustness to Adversarial Examples through an Ensemble of Specialists, Christian Gagné et al. explored this using an ensemble of "specialist" models trained on specific subsets of confusing classes. When faced with an adversary, the specialists fail to reach a consensus, causing a measurable drop in overall confidence and a spike in predictive entropy. Mahdieh Abbasi expanded on this in Toward Adversarial Robustness by Diversity in an Ensemble of Specialized Deep Neural Networks, using a fixed confidence threshold to reject attacks based on this consensus failure.

To maximize this effect, Ali Dabouei regularized the gradient diversity within ensembles in Exploiting Joint Robustness to Adversarial Perturbations, ensuring that adversarial perturbations trigger wildly different responses across member models, thereby amplifying the detection signal.

Clean Input   ───► [ Ensemble of Models ] ───► Low Variance  ───► Accept
Adversarial   ───► [ Ensemble of Models ] ───► High Variance ───► Reject (Spike)

This concept is also highly effective for unsupervised anomaly detection. For instance, in Uninformed Students: Student-Teacher Anomaly Detection With Discriminative Latent Embeddings, the authors used the predictive variance of a student ensemble to localize pixel-precise anomalies. Similarly, Misha Glazunov et al. showed in Do Bayesian variational autoencoders know what they don't know? that the variance between sampled decoders in a Bayesian VAE reliably flags unfamiliar data, even when the average likelihood remains high.

Rather than just passively measuring this gap, Anh Bui et al. explicitly trained ensembles to be maximally uncertain on adversarial examples while remaining highly confident on natural data in Improving Ensemble Robustness by Collaboratively Promoting and Demoting Adversarial Robustness, creating an intentional, high-precision entropy gap for threat detection.

The Trojan Signature: Unnatural Prediction Stability

While adversarial examples cause uncertainty to increase, backdoor (or Trojan) attacks exhibit the exact opposite statistical signature: abnormally low uncertainty under perturbation.

In a backdoor attack, an adversary poisons the training data so that the model behaves normally on clean inputs but immediately outputs a specific target class whenever a secret "trigger" (like a small patch or a specific sentence structure) is present. Because neural networks naturally memorize simple, highly predictive shortcuts faster than complex legitimate features, the trigger establishes an incredibly robust, redundant neural pathway. As noted by Rui Wang et al. in The Victim and The Beneficiary: Exploiting a Poisoned Model to Train a Clean Model on Poisoned Data and Zonghao Ying et al. in DLP: towards active defense against backdoor attacks with decoupled learning process, samples exhibiting low entropy or high confidence during the initial training phase are highly correlated with the presence of these simple backdoor triggers.

At runtime, this creates a unique vulnerability for the attacker: the model's prediction becomes completely invariant to the rest of the input. If you take a suspected image and heavily perturb it—by blending it with random noise or overlaying other images—a benign image will cause the model's predictions to fluctuate wildly, yielding high prediction entropy. However, if the image contains a backdoor trigger, the trigger continues to dominate the network's internal representations, forcing the target class prediction with unwavering certainty and abnormally low entropy.

Benign Input  + Noise ───► [ Model ] ───► Fluctuating Predictions ───► High Entropy
Trojan Input  + Noise ───► [ Model ] ───► Rigid Target Class      ───► Low Entropy

Yansong Gao et al. formalized this test-time input perturbation defense in STRIP: A Defence Against Trojan Attacks on Deep Neural Networks, using Shannon entropy filtering to isolate Trojaned inputs. This principle generalizes across modalities and architectures:

Internal vs. External Perturbations

Instead of perturbing the input externally, we can perturb the network internally to achieve a similar diagnostic effect. Wei Li introduced this concept in PSBD: Prediction Shift Uncertainty Unlocks Backdoor Detection, which analyzes prediction instability under dropout. Because backdoor triggers rely on highly redundant, robust neural paths, their predictions stubbornly resist dropout disruption. Clean samples, relying on fragile and distributed feature paths, fluctuate significantly. Measuring this "prediction shift uncertainty" provides a clean, high-fidelity signal to separate poisoned data from benign data.

On the defensive side of federated learning, Jung Wuk Park et al. demonstrated in Sageflow: Robust Federated Learning against Both Stragglers and Adversaries that this entropy-based filtering can be applied inversely at the server level. By evaluating the prediction entropy of incoming client weights against a small public dataset, malicious or poisoned model updates can be immediately identified and discarded before they corrupt the global model. Similarly, in physical signal processing, P. Ai et al. showed in Universal uncertainty estimation for nuclear detector signals with neural networks and ensemble learning that monitoring spikes in predictive variance generated by a model ensemble allows the system to automatically flag or discard corrupted signal samples.

Whether dealing with the high-entropy chaos of an adversarial attack or the low-entropy rigidity of a backdoor trigger, the core insight remains the same: deterministic outputs hide the true nature of a model's relationship with its data. By treating neural networks as probabilistic systems—either through input noise, ensembles, or internal weight stochasticity—we turn predictive uncertainty from a passive metric into an active, highly sensitive shield against manipulation.

Key Foundational Work

STRIP: A Defence Against Trojan Attacks on Deep Neural Networks (2019) — contributed 3 concept elements to this direction.
Dropout Inference in Bayesian Neural Networks with Alpha-divergences (2017) — contributed 2 concept elements to this direction.
Robustness to Adversarial Examples through an Ensemble of Specialists (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_uncertainty_based_adversarial_detection_9c58a11b,
  title = {uncertainty-based adversarial detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uncertainty-based-adversarial-detection/rdr_9c58a11bf30b444e99d6544b0a3a2079},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_uncertainty_based_adversarial_detection_9c58a11b_v1,
  title = {uncertainty-based adversarial detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uncertainty-based-adversarial-detection/rdr_9c58a11bf30b444e99d6544b0a3a2079/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake