The Security Risks of Explainable Machine Learning

445 concepts190 papers14 proposals

Explainable AI (XAI) promises to transform machine learning from an opaque black box into a transparent, auditable ecosystem. If a model denies a loan application, a counterfactual explanation can tell the user exactly what features to change for a favorable outcome. If a medical model flags an anomaly, a saliency map highlights the exact pixels driving the decision. This transparency is increasingly viewed as a prerequisite for trust, safety, and compliance with regulations like the EU AI Act.

However, a growing body of research reveals a fundamental paradox: explanations are information leaks. By revealing why a model made a specific decision, an explanation inadvertently exposes the model's internal geometry, decision boundaries, and training data distribution. The very utility that makes explanations valuable to honest users makes them a powerful blueprint for adversaries.

The Geometry of Recourse: Counterfactuals as Adversarial Roadmaps

To understand why explanations introduce security risks, consider Counterfactual Explanations (CFEs). A CFE provides algorithmic recourse by calculating the minimal perturbation required to change a model's output (e.g., "If your income were \$5,000 higher, your loan would be approved").

Now consider an Adversarial Example (AE). An AE is a malicious input crafted by calculating the minimal perturbation required to change a model's output (e.g., tweaking an image of a stop sign so an autonomous vehicle misclassifies it as a speed limit sign).

Mathematically, these two concepts are deeply intertwined. As Chirag Agarwal et al. demonstrate in Exploring Counterfactual Explanations Through the Lens of Adversarial Examples: A Theoretical and Empirical Analysis, CFEs and AEs are functionally indistinguishable in terms of the optimization paths they follow. When an AI system provides a user with actionable recourse, it is simultaneously handing an attacker a precise vector pointing directly at the model's decision boundary.

   Black-Box Decision Boundary
              │
   [Class A]  │  [Class B]
              │
      X ──────┼─────────> X'
   (Input)    │      (Counterfactual /
              │       Adversarial Target)
              │

This geometric exposure makes structured models highly vulnerable to reverse-engineering. For example, Awa Khouna et al. developed the Tree Reconstruction Attack (TRA) in From Counterfactuals to Trees: Competitive Analysis of Model Extraction Attacks. They proved that because counterfactuals reveal exact feature thresholds along the decision boundary, an adversary can systematically clone proprietary tree-based models (like Random Forests) with 100% accuracy using orders of magnitude fewer queries than standard black-box attacks.

As Chudi Zhong et al. observe in Models That Are Interpretable But Not Transparent, faithful explanations typically reveal broad rules or exact distances to the decision boundary. Attackers can leverage these rules as high-quality labels for entire subspaces of the input data, drastically accelerating the process of cloning the model.

High-Dimensional Leaks: Why Richer Explanations Cost More Privacy

A common intuition is that moving away from low-level feature attributions (like pixel heatmaps) toward high-level, human-understandable concepts might protect system privacy. However, the opposite is true: richer, more structured explanations contain a higher signal-to-noise ratio regarding the model's internal state.

In Translating theory into practice: assessing the privacy implications of concept-based explanations for biomedical AI, Adriano Lucieri et al. evaluate how concept-based explanations affect data privacy. While a pixel-level heatmap provides noisy, localized information, concept-based explanations operate in higher-dimensional spaces that describe internal model logic through structured categories. This extra structural information gives adversaries a clean signal to execute Membership Inference Attacks (MIAs)—allowing them to determine with high certainty whether a specific individual's sensitive data was used to train the model.

Similarly, Neel Patel et al. show in High Dimensional Model Explanations: an Axiomatic Approach that exposing complex, high-dimensional feature interactions significantly increases the risk of training data leakage. The unique ways features interact act as a fingerprint for the underlying training samples, expanding the system's attack surface.

Shrinking the Version Space: The Mechanics of Model Extraction

To understand why explanations accelerate model stealing, it is helpful to use the learning-theoretic concept of a version space—the set of all possible models consistent with the observed data.

When an attacker queries a purely black-box API, they only receive a label (e.g., "Approved" or "Denied"). The version space of models that could produce that label remains massive. The attacker must send millions of random queries to slowly narrow down the possibilities and clone the model.

However, when the API returns both a label and an explanation, the version space collapses. In XAudit : A Learning-Theoretic Look at Auditing with Explanations, Chhavi Yadav et al. prove that high-fidelity explanations drastically reduce query complexity. The structural information contained in an explanation eliminates millions of candidate models simultaneously, sometimes reducing the query budget required to extract proprietary model weights down to a single interaction.

   Pure Black-Box Query:                  Explanation-Guided Query:
   Eliminates a single point.             Eliminates an entire subspace.

   ┌─────────────────────────┐            ┌─────────────────────────┐
   │ ░░░░░░░░░░░░░░░░░░░░░░░ │            │ ██████████░░░░░░░░░░░░░ │
   │ ░░░░░░░░░░░░░░░░░░░░░░░ │            │ ██████████░░░░░░░░░░░░░ │
   │ ░░░░░░░░░░X░░░░░░░░░░░░ │            │ ██████████░░░░░░░░░░░░░ │
   │ ░░░░░░░░░░░░░░░░░░░░░░░ │            │ ██████████░░░░░░░░░░░░░ │
   └─────────────────────────┘            └─────────────────────────┘
   (Large Version Space Remaining)        (Version Space Drastically Reduced)

This vulnerability spans across modalities. In the domain of graph data, Bin Ma et al. demonstrate in How Explanations Leak the Decision Logic: Stealing Graph Neural Networks via Explanation Alignment that an adversary can exploit the subgraph explanations of Graph Neural Networks (GNNs). By aligning the rank of important features provided by the explanation layer, attackers can train a surrogate model that mirrors the proprietary logic of the target system with minimal queries.

The Governance and Technical Deadlock

The tension between transparency and security creates a difficult challenge for AI governance and system design.

On one hand, developers can implement active defenses to protect their intellectual property. For example, to prevent competitors from stealing large language models via knowledge distillation, Pingzhi Li et al. propose DOGe in DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation. This method subtly obfuscates the model's reasoning traces to make them un-learnable for student models. Similarly, in XRand: Differentially Private Defense against Explanation-Guided Attacks, Truc Nguyen introduces a defense that applies local differential privacy to randomize feature rankings, preventing attackers from identifying critical vulnerabilities.

However, these defenses inherently reduce the clarity, faithfulness, and auditability of the explanations, making them less useful for legitimate users or third-party auditors who require deep access to identify systemic risks.

On the other hand, attempting to secure a system by restricting its outputs can run afoul of legal mandates. Restricting an API to return only binary decisions (e.g., a simple "yes" or "no" without any confidence scores or explanations) is an effective way to mitigate black-box adversarial probing. Yet, as Sunpill Kim et al. point out in Non-Adaptive Adversarial Face Generation, this defense strategy directly conflicts with emerging legal frameworks like the EU AI Act, which explicitly mandate detailed, explainable outputs for high-stakes AI systems.

As machine learning models are deployed in increasingly regulated and adversarial environments, resolving this tension remains an open problem. Designers cannot treat explainability and security as independent goals; instead, they must treat transparency as a cryptographic exposure that must be carefully budgeted.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 210 papers
  2. Version 1 · 2026-10-04 · 205 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_transparency_security_tradeoff_0d582642,
  title = {transparency-security tradeoff},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transparency-security-tradeoff/rdr_0d582642e52c49d1bb635b794ecef41d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_transparency_security_tradeoff_0d582642_v2,
  title = {transparency-security tradeoff},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transparency-security-tradeoff/rdr_0d582642e52c49d1bb635b794ecef41d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake