Features of Explainability: How users understand counterfactual and causal explanations for categorical and continuous features in XAI

Greta Warren, Mark T. Keane, Ruth M. J. Byrne

TL;DR

This paper investigates how users understand AI decisions based on counterfactual vs. causal explanations and whether categorical features are easier to grasp than continuous ones. They found that while both explanation types improve prediction accuracy over no explanation, counterfactuals are subjectively preferred, and categorical features lead to better understanding, especially over time.

What Problem Does This Paper Address?

As AI systems become more prevalent in decision-making, understanding why they make certain choices is crucial for trust, fairness, and recourse. Explainable AI (XAI) aims to provide these explanations. Counterfactual explanations, which tell users "if X had been different, the outcome would have changed," are a popular XAI technique. However, there's a lack of large-scale studies comparing their effectiveness to other explanation types, like causal explanations ("X happened because of Y"), and how they perform with different types of data features (categorical vs. continuous). This paper addresses this gap by examining how users comprehend AI decisions when presented with counterfactual and causal explanations, specifically focusing on how the nature of the features (categorical or continuous) influences this understanding.

Methods

The researchers conducted a controlled user study with 127 participants. They developed a simulated AI system, "SafeLimit," designed to predict whether individuals would be over or under the legal blood alcohol content limit for driving, based on five features: units of alcohol, weight, duration of drinking, gender, and stomach fullness.

The study had two main phases:

  1. Training Phase: Participants made predictions about the AI's decisions and received feedback along with one of three explanation types:
  2. Counterfactual: "If X had been different, you would have been under the limit."
    • Causal: "You are over the limit because of X."
    • Control: A simple description of the outcome.
  3. Testing Phase: Participants made predictions on new instances without any feedback or explanations.

The study measured both objective accuracy (how often participants correctly predicted the AI's decision) and subjective judgments (satisfaction and trust in the explanations and the system). They also analyzed the impact of categorical features (gender, stomach fullness) versus continuous features (units, weight, duration).

Datasets

The study used a dataset of 2000 instances generated from the Widmark equation for blood alcohol content. For the experiment, 80 instances were randomly selected and then split into two sets of 40 for the training and testing phases. These instances were carefully constructed to create counterfactual and causal explanations by incrementally changing feature values. No new benchmark datasets were introduced.

Key Insights and Ideas

  • Explanations Matter, But How? Providing any explanation (counterfactual or causal) improves users' ability to predict the AI's decisions compared to no explanation at all. This highlights the fundamental value of explainability.
  • Counterfactuals Win Subjectively: While both explanation types boost objective accuracy, users feel better about counterfactual explanations, rating them as more satisfying and trustworthy. This suggests a potential disconnect between objective performance and user perception.
  • Categorical Features are King (for Understanding): Users consistently understand and predict decisions better when explanations involve categorical features (like gender) compared to continuous ones (like weight or units consumed). This effect becomes more pronounced with experience.
  • The Illusion of Understanding? The paper raises concerns that subjective preference for counterfactuals might lead users to feel they understand the system better than they actually do, potentially fostering inappropriate trust.

Key Results

  • Objective Accuracy:
    • Counterfactual explanations led to slightly higher accuracy than causal explanations, which in turn were slightly better than the control group.
  • Participants were significantly more accurate when predicting decisions based on categorical features (gender, stomach fullness) than continuous features (units, weight, duration) in both training and testing phases.
  • Accuracy for categorical features improved from the training to the testing phase, suggesting learning, while accuracy for continuous features remained stable.
  • Subjective Measures:
    • Participants reported significantly higher satisfaction with counterfactual explanations compared to causal explanations.
  • Participants reported significantly higher trust in counterfactual explanations compared to causal explanations.
  • Both satisfaction and trust were generally higher for counterfactuals and descriptions than for causal explanations.

Limitations and Future Work

  • Feature Diversity: The study used binary categorical features. Future work could explore the impact of categorical features with more than two values to see if there's a limit to human comprehension.
  • Actionability vs. Simplicity: The findings suggest that the simplicity of categorical features (e.g., two distinct values) might be more important for understanding than abstract concepts like mutability or actionability, which are often prioritized in XAI methods.
  • Semi-Factual Explanations: The paper suggests exploring "semi-factual" explanations (which describe how a feature could be changed without altering the outcome) as another avenue for XAI research.
  • Ethical Implications: The authors highlight the ethical concern that subjective preference for explanations might not always correlate with genuine understanding, potentially leading to overconfidence and misplaced trust. Further research is needed to explore this "illusion of explanatory depth."

Summary updated

Abstract

Counterfactual explanations are increasingly used to address interpretability, recourse, and bias in AI decisions. However, we do not know how well counterfactual explanations help users to understand a systems decisions, since no large scale user studies have compared their efficacy to other sorts of explanations such as causal explanations (which have a longer track record of use in rule based and decision tree models). It is also unknown whether counterfactual explanations are equally effective for categorical as for continuous features, although current methods assume they do. Hence, in a controlled user study with 127 volunteer participants, we tested the effects of counterfactual and causal explanations on the objective accuracy of users predictions of the decisions made by a simple AI system, and participants subjective judgments of satisfaction and trust in the explanations. We discovered a dissociation between objective and subjective measures: counterfactual explanations elicit higher accuracy of predictions than no-explanation control descriptions but no higher accuracy than causal explanations, yet counterfactual explanations elicit greater satisfaction and trust than causal explanations. We also found that users understand explanations referring to categorical features more readily than those referring to continuous features. We discuss the implications of these findings for current and future counterfactual methods in XAI.
BibTeX
@inproceedings{warren2022features,
  author = {Greta Warren and Mark T. Keane and Ruth M. J. Byrne},
  title = {{Features of Explainability: How users understand counterfactual and causal explanations for categorical and continuous features in XAI}},
  booktitle = {CAKR @ IJCAI},
  year = {2022},
  url = {https://openreview.net/forum?id=5s158UJMh7},
}
Figure