Detecting Potential Local Adversarial Examples for Human-Interpretable Defense

Xavier Renard, Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Marcin Detyniecki

Introduction

As machine learning (ML) models are increasingly deployed in high-stakes domains like finance and insurance, they become targets for evasion attacks. In these scenarios, malicious actors subtly manipulate input features—such as age or income on a credit application—to flip a model's decision from "rejected" to "approved."

Traditional adversarial defenses often focus on global model robustness or anomaly detection in image data, which are difficult to apply to tabular data where "fake" values are not easily distinguishable. Furthermore, many existing defenses act as "black boxes," providing no explanation for why an input was flagged. This paper introduces Local Adversarial Detection (LAD), a model-agnostic defense mechanism designed to bridge the gap between security and interpretability by identifying specific features likely to have been manipulated.

Methodology

The LAD algorithm focuses on the local neighborhood of a specific suspicious input ($x$) rather than attempting to explain the entire complex model. The process follows four key steps:

  1. Support Point Generation: The algorithm generates random points in the feature space that have a different classification than the input $x$.
  2. Boundary Search: For each random point, LAD performs a line search to find the exact "touchpoint" where the model’s decision flips. This identifies the closest decision boundaries to the input.
  3. Local Surrogate Training: A shallow, interpretable Decision Tree is trained on points generated around these touchpoints. This surrogate acts as a simplified local map of the complex model's behavior.
  4. Feature Importance Extraction: By extracting feature importance scores from the Decision Tree, LAD identifies which features the model is most sensitive to at that specific location. These features are the primary candidates for adversarial manipulation.
The boundary search process identifies touchpoints by searching along line segments between the input and random points of different classes.

Experimental Evaluation

The researchers evaluated LAD using the German Credit dataset, which contains 1,000 credit applications. They trained a Random Forest (200 trees) as the target classifier. To test the defense, they simulated evasion attacks where users manipulated specific features to gain approval. The goal was to determine if LAD could accurately approximate the Random Forest's local logic and flag the manipulated features.

Key Results and Interpretability

LAD demonstrated significantly higher Local Fidelity compared to existing substitute model methods. This means the simplified local explanation matched the complex model's actual behavior much more closely in the immediate vicinity of the input.

In a practical test case, a user manipulated their age (changing it from 47 to 46) to secure credit approval. LAD successfully flagged "Age in years" as a top critical feature, assigning it a high importance score. This provides a clear, actionable signal for a human expert to prioritize verifying that specific piece of information.

The local surrogate model (represented by the black line) provides a simplified, interpretable approximation of the complex decision boundary near the input.

Conclusion and Future Work

LAD offers a promising path toward human-in-the-loop adversarial defense. By treating interpretability as a shield, it empowers experts to understand how a decision could be changed, rather than just flagging it as suspicious.

As this is preliminary work, future research will focus on scaling the algorithm to more complex datasets, optimizing the computational overhead of the boundary search phase, and exploring the trade-offs between the number of flagged features and defense accuracy.

Summary updated

BibTeX
@inproceedings{renard2018detecting,
  author = {Xavier Renard and Thibault Laugel and Marie-Jeanne Lesot and Christophe Marsala and Marcin Detyniecki},
  title = {{Detecting Potential Local Adversarial Examples for Human-Interpretable Defense}},
  booktitle = {Nemesis/UrbReas/SoGood/IWAISe/GDM@PKDD/ECML},
  year = {2018},
  doi = {10.1007/978-3-030-13453-2\_4},
}
Figure