TL;DR
This paper investigates what makes counterfactual explanations satisfying to users by analyzing human ratings of various explanatory qualities. They found that Feasibility (how actionable the suggested changes are) and Trust (belief in the outcome) are the strongest predictors of user satisfaction, even more so than overall satisfaction ratings. This work provides a data-driven approach to building better, more human-centric AI explanations.
What Problem Does This Paper Address?
Explainable AI (XAI) aims to make AI decisions understandable, and counterfactual explanations are a popular method. These explanations show users how small changes to input data could lead to a different outcome. However, evaluating the quality of these explanations from a human perspective is challenging. Traditional metrics like "how close is the counterfactual to the original?" or "how few features changed?" don't always align with what users actually find helpful or satisfying. User studies are the gold standard but are expensive and hard to scale. Furthermore, simply asking users for an overall satisfaction score doesn't tell us why they are satisfied or dissatisfied. This paper addresses this gap by modeling overall user satisfaction based on multiple, specific explanatory qualities, aiming to understand which qualities are most important and how to predict satisfaction.
Methods
The researchers used a dataset of human evaluations of counterfactual explanations. They employed:
- Exploratory Analysis: Visualizing per-metric distributions grouped by satisfaction levels to see how different qualities correlate with user satisfaction.
- Bi-clustering: Simultaneously clustering participants and scenarios to identify patterns in how groups of people evaluate explanations and how certain explanations are consistently rated.
- Factor Analysis: Investigating the underlying structure of the seven explanatory metrics to see if they can be grouped into broader factors.
- Machine Learning Models:
- Regression: Using Ordinary Least Squares (OLS) to predict overall satisfaction as a continuous score.
- Classification: Using Logistic Regression, Decision Trees, and Random Forests to predict satisfaction into discrete categories (low, medium, high).
- SHAP (SHapley Additive Explanations) Analysis: To understand the contribution of each explanatory metric to individual predictions, both globally and for specific scenarios.
Datasets
The study uses the CounterEval dataset [9], which was collected from 206 human participants who rated 30 diverse counterfactual scenarios across 8 metrics: Overall Satisfaction, Feasibility, Consistency, Completeness, Trust, Fairness, Complexity, and Understandability. This dataset is a benchmark for counterfactual explanation evaluation.
Key Insights and Ideas
- Feasibility and Trust Reign Supreme: These two qualities consistently emerged as the most powerful predictors of user satisfaction, far more than any other single metric. This aligns with common sense: users want explanations that are actionable and that they believe will actually work.
- Completeness Matters, But is Secondary: While Feasibility and Trust are dominant, Completeness also plays a significant role in user satisfaction. This suggests that users appreciate detailed explanations, provided they are also feasible and trustworthy.
- Other Metrics Still Explain a Lot: Even when Feasibility and Trust are removed, the remaining metrics (Completeness, Consistency, Fairness, Complexity, Understandability) still explain a substantial 58% of the variance in satisfaction. This highlights that a holistic view of explanation quality is important.
- Complexity is Nuanced: Complexity doesn't necessarily decrease satisfaction. Explanations can be somewhat complex and still be well-received if they are actionable and believable.
- Holistic Perception: Factor analysis suggests that users often perceive explanation quality holistically, with Feasibility, Trust, Completeness, and Consistency loading together on a primary factor.
- Demographics Influence Ratings: Machine Learning experience and Medical background significantly influenced how participants clustered and evaluated explanations, suggesting that domain expertise plays a role in what constitutes a "good" explanation.
Key Results
- OLS Regression: Achieved an R² of 0.757 ± 0.008, indicating that the explanatory metrics collectively explain a substantial portion of satisfaction variance. Feasibility (β ≈ 0.358) and Trust (β ≈ 0.362) had the largest positive coefficients.
- Omitting Feasibility and Trust: Reduced the model's R² to 0.580 ± 0.081, underscoring their critical importance. However, even without them, Completeness became the strongest driver, and the model still captured 58% of variance.
- Classification Models: Decision Tree achieved the highest accuracy (≈ 0.78) in the random split but was less robust on unseen scenarios. Logistic Regression and Random Forest showed comparable, more robust performance (≈ 0.75-0.76 accuracy).
- SHAP Analysis: Confirmed Feasibility and Trust as dominant predictors globally and locally. For specific scenarios, the model's predictions were influenced by a combination of metrics, with Feasibility and Trust often being the primary drivers.
Limitations and Future Work
- Scenario Diversity: The 30 scenarios, while varied, may not cover the full spectrum of real-world counterfactuals, especially concerning fairness or domain-specific constraints.
- Participant Backgrounds: The participant group, while large, might not fully represent specialized user groups or domain experts (e.g., clinicians, ML engineers) who might value certain metrics differently.
- Online Study Limitations: Inherent limitations of online studies, such as ensuring participants are not distracted, exist.
- Fairness and Clarity: The study did not systematically vary fairness or clarity, potentially underrepresenting their roles.
Future Work: * Gathering datasets for specialized applications and with different levels of expert knowledge. * Examining more diverse or extreme variations in explanatory factors. * Validating findings in other domains (medical, financial) and with specialized user cohorts. * Refining how explanatory metrics are measured and optimized for diverse audiences.
Summary updated