The effect of data poisoning on counterfactual explanations

André Artelt, Shubham Sharma, Freddy Lécué, Barbara Hammer

TL;DR

This paper reveals a critical vulnerability in "recourse" AI: counterfactual explanations can be manipulated via data poisoning. By injecting a small amount of malicious data into a training set, an attacker can significantly increase the "cost of recourse," making it intentionally harder or more expensive for users to achieve a favorable outcome (like getting a loan approved) without significantly damaging the model's overall accuracy.

What Problem Does This Paper Address?

Counterfactual Explanations (CFEs) are the "gold standard" for providing actionable recourse in XAI (Explainable AI). If a bank’s AI rejects your loan, a CFE tells you: "If your income were $5,000 higher, your loan would be approved."

While previous research has shown that these explanations can be manipulated at "run-time" (inference), this paper looks at the training stage. The authors ask: Can an attacker slip "poisonous" samples into the training data to shift the model's decision boundary in a way that makes these explanations much more difficult for the user to follow? This is a major concern for fairness and safety, especially in sensitive domains like finance, healthcare, and critical infrastructure.

Methods

The authors propose a model-agnostic data poisoning framework (Algorithm 1) designed to increase the cost of recourse. Unlike "label flipping" (which just tries to tank model accuracy), this attack is surgical.

  1. Targeting: The attack can be Global (affecting everyone), Sub-group (targeting specific protected classes), or Local (targeting a single individual).
  2. Poison Generation: The attacker identifies samples near the decision boundary and uses a counterfactual generation method (like DiCE) to create "poisonous" versions of these samples. These samples are placed just across the decision boundary.
  3. Stealth: The optimization objective ensures these poisonous samples are "realistic"—they sit on the data manifold and look like legitimate data points, making them hard to detect.
  4. Mechanism: By adding these points, the attacker forces the newly trained model to shift its decision boundary further away from the "unfavorable" class, effectively "pushing the goalposts" back for the user.

Datasets

The study evaluates the attack on three standard ML benchmarks and one real-world case study:

  • Diabetes: Predicting disease progression.
  • Communities & Crime: Predicting crime rates (used to test sub-group fairness).
  • German Credit: Loan approval (used to test the cost of financial recourse).
  • Hanoi Benchmark (WDN): A simulated Water Distribution Network used to detect and localize sensor faults. This newly applied context shows how poisoning can hinder engineers from identifying the cause of an anomaly.

Key Insights and Ideas

  • Recourse as a Target: Most poisoning attacks target accuracy or fairness. This paper highlights that the explanation itself is a high-value target for malicious actors.
  • The "Plausibility" Trap: Because the poisonous samples are generated to be "plausible" (close to existing data), they bypass traditional outlier detection. The attack doesn't look like "noise"; it looks like a subtle shift in the underlying distribution.
  • Ineffectiveness of Label Flipping: The authors show that classic label flipping (randomly changing training labels) has almost no effect on the cost of recourse. This proves that attacking XAI requires specialized, "informed" poisoning strategies.
  • Saturating Effects: There is an upper bound to the damage. Increasing the poisoning budget (e.g., from 5% to 70%) eventually leads to diminishing returns in how much the recourse cost can be increased.

Key Results

  • Significant Cost Increase: Injecting just 5% poisonous data led to a statistically significant increase in recourse cost across almost all scenarios. In some local attack cases on the Diabetes dataset, the cost of recourse jumped by 39%.
  • Increased Sparsity: Poisoning doesn't just make the "distance" to a favorable outcome longer; it often forces the user to change more features (lower sparsity) to get a different result.
  • Defense Failure: The authors tested five classic defenses, including Isolation Forest and k-NN defense. These methods struggled immensely, often showing a recall of less than 20-30%, meaning the vast majority of poisonous samples remained in the training set undetected.
  • Real-world Impact: In the water network case study, poisoning the event detector made it significantly more difficult for operators to pinpoint which sensor was actually failing, potentially delaying critical repairs.

Limitations and Future Work

  • Attacker Knowledge: The attack assumes the attacker has access to a prediction interface and some method for generating counterfactuals to create the poison. While "black-box," it still requires a degree of interaction with a proxy model.
  • Untargeted Side Effects: In local attacks, the authors noted that "untargeted" individuals sometimes saw a slight increase in their recourse costs as well. Refining the attack to be even more surgical is a suggested direction.
  • Robustness by Design: The paper concludes with a call to action: the research community needs to design CFE algorithms that are inherently robust to training data manipulations, rather than relying on post-hoc data sanitization which currently fails.

Summary updated

Abstract

Counterfactual explanations are a widely used approach for examining the predictions of black-box systems. They can offer the opportunity for computational recourse by suggesting actionable changes on how to alter the input to obtain a different (i.e., more favorable) system output. However, recent studies have pointed out their susceptibility to various forms of manipulation. This work studies the vulnerability of counterfactual explanations to data poisoning. We formally introduce and investigate data poisoning in the context of counterfactual explanations for increasing the cost of recourse on three different levels: locally for a single instance, a sub-group of instances, or globally for all instances. In this context, we formally introduce and characterize data poisonings, from which we derive and investigate a general data poisoning mechanism. We demonstrate the impact of such data poisoning in the critical real-world application of explaining event detections in water distribution networks. Additionally, we conduct an extensive empirical evaluation, demonstrating that state-of-the-art counterfactual generation methods and toolboxes are vulnerable to such data poisoning. Furthermore, we find that existing defense methods fail to detect those poisonous samples.
BibTeX
@article{artelt2026effect,
  author = {André Artelt and Shubham Sharma and Freddy Lécué and Barbara Hammer},
  title = {{The effect of data poisoning on counterfactual explanations}},
  journal = {Information Fusion},
  year = {2026},
  doi = {10.1016/j.inffus.2026.104237},
}

Loading related content...

Figure