Analyzing the Influence of Training Samples on Explanations

André Artelt, Barbara Hammer

Abstract

Explainable AI (XAI) has become a cornerstone for interpreting the decision-making processes of complex AI systems, particularly through counterfactual explanations that provide users with a path to recourse. However, when these explanations are unexpected, prohibitively expensive, or biased, users lack the tools to understand the underlying cause within the training data. This paper addresses this gap by investigating the influence of individual training samples on model explanations rather than on the model's predictive performance alone. We propose a novel algorithm designed to identify influential training samples that drive specific explanation properties, such as the cost of recourse or fairness disparities between protected groups. Our findings demonstrate that the data points shaping a model's logic are often distinct from those ensuring its accuracy, offering a new avenue for debugging and refining AI behavior through targeted data pruning.

Introduction

As AI systems are increasingly deployed in high-stakes domains like finance and healthcare, the need for transparency has led to the rise of Explainable AI (XAI). A popular method within XAI is counterfactual recourse, which informs a user how to change their features to achieve a desired outcome (e.g., "If you increase your savings by $5,000, your loan will be approved").

While these explanations clarify how a model arrived at a decision, they do not explain why the model learned that specific logic from its training data. This lack of transparency is problematic when explanations suggest impractical actions or reveal systemic biases. This research introduces a framework to trace these undesirable explanation properties back to specific training samples, effectively providing an "explanation of the explanation."

The Problem: When Recourse is Unfair or Costly

Counterfactual explanations are intended to be actionable. However, they often suffer from two major issues:

  1. High Cost: The suggested changes might be prohibitively difficult for a user to implement.
  2. Fairness Gaps: The "cost" of achieving a positive outcome may vary significantly between demographic groups (e.g., based on gender or age), leading to systemic inequality.

Existing data valuation techniques, such as Data-SHAP, focus on how samples affect a model's accuracy. This paper argues that accuracy is an insufficient metric for debugging explanations, as the samples that make a model accurate are not necessarily the ones that define its internal reasoning or the resulting recourse paths.

Methodology

To identify influential samples, the authors adapted the concept of Data Valuation to the realm of explanations. The core challenge is the computational cost; retraining a model for every possible subset of data to see how an explanation changes is infeasible.

Gradient-Based Monte-Carlo Approach

The researchers proposed an algorithm that estimates "influence scores" ($\phi_i$) on the fly during the training process. By tracking how a sample's gradient update shifts the model’s internal logic regarding explanations, the algorithm can estimate influence without exhaustive retraining.

Logit Approximation

Calculating actual counterfactuals at every training step is computationally expensive. To solve this, the authors utilized a logit approximation. By measuring the difference in "logits" (raw scores before the final decision layer), they created a proxy for the distance to the decision boundary. This allows for a feasible calculation of how much a specific training point moves that boundary for a given user.

Empirical Evaluation

The approach was tested on two standard fairness benchmarks:

  • Diabetes Dataset: Used to predict disease progression across 442 patients.
  • German Credit Dataset: A 1,000-sample dataset for loan approval, known for inherent biases regarding the "sex" attribute.

Key Findings

  1. Accuracy vs. Logic: The study confirmed that samples influential for model accuracy are often different from those influential for model explanations. Standard data valuation tools are therefore inadequate for debugging XAI.
  2. Targeted Pruning: By identifying and removing samples that drive up the cost of recourse, the researchers could "fix" the model's explanations. Removing these influential points reduced the fairness gap and the average cost of recourse more effectively than random pruning.
  3. Performance Maintenance: Unlike baseline methods that prioritize predictive performance, this method maintained a high F1-score while successfully improving the quality and fairness of the explanations.

Discussion and Limitations

This work represents a significant step toward making AI systems more accountable by linking model behavior back to the data source. However, several limitations remain:

  • Approximation Limits: The logit-based proxy, while efficient, lacks formal guarantees for all model architectures.
  • Collective Influence: The current framework evaluates samples individually, potentially missing "interaction effects" where groups of samples collectively influence an explanation.
  • Scalability: While optimized, calculating influence scores for massive, industrial-scale datasets remains a computational challenge.

Conclusion

By shifting the focus from feature space to data space, this research provides a vital tool for developers to audit their models. Identifying the training samples responsible for unfair or impractical explanations allows for targeted data cleaning, ensuring that AI systems are not only accurate but also provide equitable and actionable guidance to their users.

Summary updated

Abstract

EXplainable AI (XAI) constitutes a popular method to analyze the reasoning of AI systems by explaining their decision-making, e.g. providing a counterfactual explanation of how to achieve recourse. However, in cases such as unexpected explanations, the user might be interested in learning about the cause of this explanation -- e.g. properties of the utilized training data that are responsible for the observed explanation. Under the umbrella of data valuation, first approaches have been proposed that estimate the influence of data samples on a given model. In this work, we take a slightly different stance, as we are interested in the influence of single samples on a model explanation rather than the model itself. Hence, we propose the novel problem of identifying training data samples that have a high influence on a given explanation (or related quantity) and investigate the particular case of differences in the cost of the recourse between protected groups. For this, we propose an algorithm that identifies such influential training samples.
BibTeX
@misc{artelt2024analyzing,
  author = {André Artelt and Barbara Hammer},
  title = {{Analyzing the Influence of Training Samples on Explanations}},
  year = {2024},
  eprint = {2406.03012},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.03012},
}

Loading related content...

Figure