When deep neural networks are deployed in high-stakes environments, their vulnerability to adversarial manipulation and distribution shifts becomes a critical liability. An adversarial perturbation can cause a model to misclassify an image or text with near-perfect confidence while remaining completely imperceptible to human observers. Traditional defenses often attempt to inspect raw inputs for anomalies or calibrate output confidence scores, but both surfaces can be easily bypassed or manipulated by sophisticated adversaries.
To build more resilient systems, a growing body of research shifts the defensive focus inward. Instead of asking what a model decided, researchers are monitoring why a model made a decision. By treating internal feature attributions—such as SHAP values, LIME scores, or gradient-based saliency maps—as a diagnostic signature, we can detect adversarial attacks, data poisoning, and out-of-distribution data through the anomalies left behind in the model's internal reasoning space.
The Intuition: Why the Explanation Space Resists Deception
To understand why feature attribution monitoring works, consider how a neural network processes a legitimate input versus an adversarial one. For a natural image of a dog, the model relies on a coherent manifold of robust, semantic features—such as the shape of the ears, the texture of the fur, and the position of the nose. An interpretability tool mapping this decision will produce a clean, concentrated attribution map centered on these logical regions.
An adversarial attack, however, works by introducing tiny, highly optimized perturbations that exploit the model's non-robust decision boundaries. To force a misclassification (e.g., turning a "dog" into an "airplane"), the attack must compel the network to ignore the dominant semantic features and instead rely on scattered, low-magnitude pixel relationships.
While the resulting input looks identical to a human, and the final output is confidently incorrect, the internal mechanics of the model experience severe friction. Because the model is forced to synthesize a decision from unnatural feature combinations, its internal "attention" becomes distorted. This distortion leaves an indelible fingerprint in the explanation space, making feature attributions an exceptionally reliable signal for anomaly detection.
Anatomy of an Attribution Anomaly
When an anomalous or malicious input forces a model to deviate from its trained logic, the resulting feature attribution maps exhibit several distinct structural changes:
- Scattered Attention and High Spatial Entropy: In computer vision models, natural inputs yield highly localized and structured saliency maps. As noted by Stadelmann et al. and Wu, adversarial perturbations shatter this coherence, causing the model's visual focus to scatter unnaturally across the input. Measuring the spatial entropy or chaotic distribution of these gradient responses provides a quantitative metric to flag compromised inputs.
- Unnatural Feature Reliance and Trigger Words: In natural language processing and tabular domains, adversarial inputs often rely on specific "trigger" tokens or non-standard features to flip a model's prediction. Singla et al. demonstrated this in automated essay scoring systems, where adversarial attempts to manipulate grades caused irrelevant trigger words to receive disproportionately high importance scores. Similarly, Watson et al. observed that imperceptible changes in medical data force models to rely on non-standard diagnostic features, creating highly visible anomalies in the resulting explanation maps.
- Internal Feature Distortion: Beyond final layer explanations, malicious queries can warp the entire distribution of internal representations. In model extraction attacks—where an adversary systematically queries a victim model to clone its functionality—the distribution of internal feature activations deviates significantly from benign usage patterns, a phenomenon termed "feature distortion" by Yao et al.
Architectures for Monitoring and Defense
To operationalize these insights, researchers have developed several distinct frameworks to monitor the explanation space, ranging from supervised auxiliary classifiers to unsupervised distribution tracking.
Supervised Attribution Classifiers
The most direct approach is to treat the explanation map itself as an image or feature vector and train a secondary model to classify it as "benign" or "malicious."
Input Data ---> [ Primary Model ] ---> Feature Attributions (SHAP/Saliency)
| |
Final Output [ Detector Model ]
|
---> Clean / Adversarial
Fidel et al. pioneered this by extracting SHAP signatures from internal neural network layers and training a secondary binary classifier on these signatures, achieving high detection generalization across unseen attack types. This strategy adapts well across modalities:
- Vision: Zhang et al. feed attribution maps from UAV vision systems directly into standard architectures like ResNet and EfficientNet to detect adversarial manipulation.
- Text: Huber et al. train model-agnostic detectors on word-level SHAP values to identify adversarial text injections.
- Time-Series/Control: Aouf et al. deploy an LSTM-based monitor over sequential SHAP values to protect autonomous spacecraft relative pose estimation systems in real time.
Unsupervised Distribution Tracking
In many real-world scenarios, explicit adversarial examples are unavailable during training, or the system must defend against silent distribution shifts where the data distribution drifts over time.
To solve this, Mougan et al. (2022) and Mougan et al. (2025) introduced the Explanation Shift framework. Instead of monitoring raw input features, which can be high-dimensional and noisy, the system tracks the statistical distribution of local feature attributions (such as SHAP or LIME values) between training and deployment. When a distribution shift or a training data poisoning attack occurs (as explored by Zhang et al. in EEG emotion evaluation systems), the explanation shift detector flags the anomaly because the model's internal decision logic has fundamentally altered, even if ground-truth labels are missing.
Risk Optimization and Robust Training
Rather than passively flagging anomalies, feature attributions can be integrated directly into optimization objectives to filter noise or robustify the model.
For geometric data like 3D point clouds, Li et al. developed PointCVaR, a framework that calculates the gradient-based attribution of each individual point to the final classification score. By minimizing the Conditional Value-at-Risk (CVaR) of these point-wise attributions, the system dynamically scrubs adversarial noise and backdoor points based on their disproportionate internal influence rather than their spatial coordinates. To handle complex, high-dimensional outputs like 3D medical segmentations, Baniecki et al. aggregate millions of voxel-level attributions into region-of-interest matrices, providing clean inputs for downstream outlier detection algorithms.
Finally, this paradigm can be used to train inherently robust models. Boopathy et al. demonstrated that forcing a network's interpretation maps to remain stable during training fundamentally improves its adversarial resistance. By minimizing a dual-class interpretation discrepancy measure, they ensure that an attacker cannot manipulate a prediction without leaving a glaring, detectable signature in the model's explanation maps.
Key Foundational Work
Deep Learning in the Wild (2018) — contributed 2 concept elements to this direction.
Why the Failure? How Adversarial Examples Can Provide Insights for Interpretable Machine Learning (2018) — contributed 1 concept elements to this direction.
When Explainability Meets Adversarial Learning: Detecting Adversarial Examples using SHAP Signatures (2019) — contributed 1 concept elements to this direction.