When Explainability Meets Adversarial Learning: Detecting Adversarial Examples using SHAP Signatures

Gil Fidel, Ron Bitton, Asaf ShabtaiZenodo 2019 · DOI · OpenAlex

TL;DR

This research bridges the gap between Explainable AI (XAI) and adversarial security by leveraging SHAP (Shapley Additive Explanations) to detect malicious inputs. By treating the importance scores of internal neural network layers as a unique "fingerprint," the authors developed a detector that identifies adversarial examples with ~97% AUC. Crucially, this method generalizes to novel attacks it has never encountered during training.

The Vulnerability of Deep Learning

Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial examples: inputs modified by human-imperceptible perturbations that cause a model to confidently misclassify them. A classic example is a computer vision system perceiving a "stop" sign as a "speed limit" sign due to a few strategically placed pixels.

Security researchers currently face a "cat and mouse" game. Most existing detection methods are brittle; they perform well against specific attacks they were trained on but fail against slightly different mathematical approaches. Furthermore, because DNNs are often treated as "black boxes," understanding why they are fooled is difficult. This paper asks: Can the tools we use to explain model decisions also be used to defend them?

Methodology: The Three-Phase Pipeline

The authors propose a robust detection framework that shifts the focus from raw data to the model's internal reasoning process.

The three-phase pipeline for creating a repository, generating signatures, and constructing the detector.
  1. Creating a Repository: A massive dataset of normal images and adversarial examples is generated using a diverse suite of state-of-the-art attacks, including FGSM, PGD, and Carlini & Wagner.
  2. Generating XAI Signatures: For every input, the system uses SHAP (Shapley Additive Explanations)—specifically the DeepExplainer variant—to calculate the contribution of each neuron in the penultimate layer (the final layer before output). These contribution scores are concatenated into a vector termed an "XAI Signature."
  3. Detector Construction: A simple binary classifier (a 3-layer neural network) is trained to distinguish between "Normal" and "Adversarial" XAI Signatures.

By focusing on the penultimate layer, the authors target high-level features that are much harder for an attacker to manipulate without destroying the intended adversarial effect.

Key Insights: Explainability as a Security Sensor

The core hypothesis is that adversarial attacks primarily manipulate "non-robust" features—patterns that are predictive for a computer but meaningless to humans. When an attack occurs, the model's "rationale" becomes anomalous.

Visual comparison of normal and adversarial examples alongside their corresponding XAI signatures, showing how signatures differ even when images look similar.

As shown above, an adversarial example (e.g., a car targeted to be classified as a cat) produces a signature that stands out from legitimate examples. Instead of looking at pixels, the detector looks at the logic of the model. If the reasoning appears corrupted, it is flagged as an attack. Because this captures an intrinsic property of adversarial examples rather than specific pixel patterns, the system generalizes effectively across different attack algorithms.

Evaluation and Results

The method was evaluated on benchmark datasets MNIST (handwritten digits) and CIFAR-10 (10 object categories) using ResNet-56 and standard CNN architectures.

  • High Performance: The detector achieved an AUC-ROC of ~0.97 for both datasets.
  • Zero-Day Robustness: In "leave-one-out" testing, where the detector faced attack types it had never seen during training, performance remained stable. This suggests the method can defend against previously unknown adversarial techniques.
  • Clustering: Dimensionality reduction (UMAP) confirmed that XAI signatures for normal data cluster neatly by class, while adversarial signatures form distinct, identifiable clusters.

Limitations and Future Outlook

While promising, the approach faces a few hurdles. Calculating SHAP values is computationally intensive, which may introduce latency in real-time applications like autonomous driving. Additionally, while SHAP is not easily differentiable—making it hard to attack—sophisticated "adaptive attackers" might eventually find ways to spoof normal-looking signatures. Future research will focus on optimizing computation and testing whether signatures can be transferred across different model architectures.

Summary updated

Abstract

State-of-the-art deep neural networks (DNNs) are highly effective in solving many complex real-world problems. However, these models are vulnerable to adversarial perturbation attacks, and despite the plethora of research in this domain, to this day, adversaries still have the upper hand in the cat and mouse game of adversarial example generation methods vs. detection and prevention methods. In this research, we present a novel detection method that uses Shapley Additive Explanations (SHAP) values computed for the internal layers of a DNN classifier to discriminate between normal and adversarial inputs. We evaluate our method by building an extensive dataset of adversarial examples over the popular CIFAR-10 and MNIST datasets, and training a neural network-based detector to distinguish between normal and adversarial inputs. We evaluate our detector against adversarial examples generated by diverse state-of-the-art attacks and demonstrate its high detection accuracy and strong generalization ability to adversarial inputs generated with different attack methods.
BibTeX
@inproceedings{fidel2019when,
  author = {Gil Fidel and Ron Bitton and Asaf Shabtai},
  title = {{When Explainability Meets Adversarial Learning: Detecting Adversarial Examples using SHAP Signatures}},
  booktitle = {Zenodo},
  year = {2019},
  doi = {10.5281/zenodo.3941782},
}

Loading related content...

Figure