TL;DR
This paper tackles the critical issue of adversarial attacks on "post-hoc explainers," which are tools used to understand how complex AI models make decisions. The authors introduce novel methods (CAD-Detect and CAD-Defend) to detect and mitigate these attacks, ensuring that AI audits are trustworthy and that AI systems aren't deceptively masking their decision-making processes.
What Problem Does This Paper Address?
Artificial intelligence (AI) is increasingly used in high-stakes domains like finance, medicine, and law. However, many AI models are "black boxes," meaning their internal workings are opaque. To address this, "post-hoc explainers" are used to interpret these models' decisions by estimating the importance of each input feature. These explainers are crucial for auditing AI systems to ensure fairness and compliance.
The problem is that these explainers themselves can be fooled. Adversaries can manipulate the AI model to produce explanations that appear benign, even if the underlying model is making biased or unfair decisions. This "scaffolding attack" makes it difficult for auditors to trust the explanations and verify that the AI system is operating ethically and legally. The paper aims to provide a robust defense against such attacks, ensuring the integrity of AI audits.
Methods
The core of the paper's contribution lies in its novel algorithms for detecting and defending against adversarial attacks on perturbation-based post-hoc explainers.
- KNN-CAD (K-Nearest Neighbors Conditional Anomaly Detection): This is a new unsupervised algorithm designed to detect anomalies in data. It works by comparing the "normal" behavior of data points to their neighbors, identifying deviations that might indicate manipulation.
- CAD-Detect: This algorithm uses KNN-CAD to detect whether an AI model is employing an adversarial attack. It compares the distribution of normal data to the distribution of data generated by the explainer, looking for significant differences that signal an attack.
- CAD-Defend: This algorithm aims to "unfool" the explainers. It works by filtering out the anomalous, potentially adversarial perturbations generated by the explainer, ensuring that the final explanation is more faithful to the true decision-making process of the AI model.
The authors also introduce new metrics to evaluate the fidelity of attacks and defenses, providing a more comprehensive way to assess their effectiveness.
Datasets
The paper evaluates its approach on three real-world, high-stakes datasets:
- COMPAS: This dataset is used for predicting recidivism risk and has been a subject of scrutiny for potential bias.
- German Credit: This dataset is used for credit scoring and is relevant for assessing fairness in financial decisions.
- Communities and Crime: This dataset combines socio-economic and crime data, used to predict crime rates.
These datasets are chosen because they contain protected attributes (like race or gender) that should ideally not influence decisions, making them suitable for testing fairness-related attacks and defenses.
Key Insights and Ideas
- Explainers are vulnerable: The paper confirms and builds upon prior work showing that even sophisticated explainers like LIME and SHAP can be tricked into providing misleading interpretations.
- Auditing requires defense: The core insight is that simply using explainers for auditing is insufficient if those explainers can be manipulated. A robust defense mechanism is necessary to ensure the trustworthiness of the audit process.
- Conditional anomaly detection is key: The paper highlights the power of conditional anomaly detection (KNN-CAD) as a foundational tool for identifying subtle deviations that indicate adversarial behavior.
- SHAP is more robust but not immune: The research suggests that SHAP's perturbation strategy is more realistic than LIME's, making it harder to attack. However, it's still susceptible, indicating that defenses are needed for even the more advanced explainers.
- Defense improves fidelity: The proposed CAD-Defend algorithm not only mitigates attacks but also helps restore the fidelity of explanations, making them more accurate representations of the AI's decision-making.
Key Results
- Successful Detection: CAD-Detect, using KNN-CAD, successfully detected adversarial attacks across all tested datasets with no false positives when using appropriate thresholds.
- Effective Defense: CAD-Defend significantly mitigated the scaffolding attack, leading to explanations that were more faithful to the true feature importance and closer to the explanations generated without an attack.
- High Fidelity Maintained: The defense mechanism achieved near-zero infidelity, meaning it did not significantly alter the explanations when no attack was present. The fidelity of the defended explanations (fidelityCAD-Defend, f) was close to the fidelity of the undefended model (fidelityf).
- Sample Efficiency: The KNN-CAD algorithm demonstrated good sample efficiency, performing well even with a small proportion of training data, which is crucial given the potential cost of querying black-box models.
Limitations and Future Work
- Data Collection Cost: A limitation noted is that collecting realistic samples for training the defense algorithm can be expensive or difficult.
- Query Costs: In real-world scenarios, querying the black-box model with perturbed samples can be costly and may raise suspicion from the auditee. The paper suggests rate-limiting queries as a practical consideration.
- Irregular Attacks: The current work doesn't explicitly consider adversaries who irregularly deploy attacks, which could be harder to detect.
- Explainer Inconsistencies: The paper acknowledges that post-hoc explainers themselves have broader issues (inconsistency, unfaithfulness, intractability). While their defense addresses adversarial manipulation, it doesn't solve all inherent problems with explainers.
- Future Work: The authors suggest exploring the ramifications of an adversarial organization being caught red-handed within existing regulatory frameworks. They also point to the need to consider the broader context of explainer limitations and potential overtrust by practitioners.
Summary updated