Explanations are a Means to an End: Decision Theoretic Explanation Evaluation

Ziyang Guo, Berk Ustun, Jessica Hullman

TL;DR

Explanations in AI are often evaluated using abstract proxies like "faithfulness," yet they frequently fail to improve real-world human-AI decision-making. This paper introduces a decision-theoretic framework that treats explanations as information signals, providing formal upper bounds and causal metrics to evaluate their actual utility on specific tasks.

What Problem Does This Paper Address?

Explainable AI (XAI) is booming, but how do we know if an explanation is actually good? Currently, researchers evaluate explanations using internal criteria (like faithfulness or robustness) or subjective user surveys. However, a recent meta-analysis of 370 studies showed that providing explanations often does not actually improve human-AI decision quality.

The core issue is a lack of theoretical grounding. Without defining the best-case potential of an explanation on a given task, it is impossible to tell whether a failed human-AI collaboration is due to a bad explanation, a poor task design, or the fact that the explanation simply contains no new useful information.

Methods

The authors frame explanation evaluation through the lens of decision theory and the value of information. They introduce three key metrics (estimands):

  1. Theoretic Value ($\Delta_\mathcal{E}$): The maximum possible performance boost an idealized, perfectly rational agent could achieve with access to the instance features.
  2. Human-Complementary Value ($\Delta_{\mathcal{E}\text{compl}}$): The marginal value the explanation offers over what a baseline human already knows or decides.
  3. Behavioral Value ($\Delta_{\mathcal{E}\text{behavioral}}$): The actual causal effect (Average Treatment Effect) observed when the explanation is given to real human decision-makers.
Fig. 1. Quantities defined in our framework. The researcher can confirm that an explanation has potential by proceeding from the theoretic to the human-complementary to the behavioral value of explanation, comparing the estimates at lower levels to those above.

The Blackwell Paradox

Crucially, the authors prove a provocative mathematical reality: for a perfectly rational Bayesian agent, an explanation adds zero value if the agent already has the raw features. This is because explanations are just a "garbling" (a deterministic function) of the features and model predictions. Therefore, explanations only have value because humans are boundedly rational—they help "unlock" information that is too cognitively expensive to extract from raw features alone.

To estimate these values on high-dimensional data (like text or images) without overfitting, the authors introduce a signal coarsening workflow (Algorithm 1) that clusters features and explanations while preserving their informational relationships. They also use V-shaped scoring rules to handle cases where the exact utility function (e.g., the cost of a false positive vs. false negative) is ambiguous.

Datasets

The framework was validated across several distinct tasks and datasets:

  • MIMIC-IV & MIMIC-CXR: Used for a medical decision-making case study (predicting cardiac dysfunction from radiographs and reports).
  • Deception Detection Dataset (Lai & Tan): Hotel reviews used to evaluate if explanations help humans detect fake reviews.
  • Sentiment Classification Dataset (Bansal et al.): Amazon Books and Beer reviews used to evaluate human-AI collaboration on sentiment analysis.
  • Jigsaw Toxic Comment Classification Dataset: Used for a mechanistic interpretability alignment audit task to detect training bias.

Key Insights and Ideas

  • Explanations assume human irrationality: If humans were perfectly rational calculators, explanations would be completely redundant with the raw data. Their true utility lies in reducing cognitive costs for boundedly rational agents.
  • The task matters more than the explanation method: The Theoretic Value of Explanation depends entirely on the decision task itself, not the specific XAI method (LIME, SHAP, etc.). If a task has a low theoretic value, no explanation method will ever make a difference.
  • Complementarity is key: An explanation can be highly informative in isolation, but if it only tells the human what they already know, its human-complementary value is zero.

Key Results

  • Deception Detection: Explanations (Heatmaps and Examples) reliably improved human performance. Baseline human accuracy was low, meaning the explanations offered high human-complementary value (~10-15% accuracy boost over AI prediction alone).
  • Sentiment Classification: Explanations failed to improve human performance. The framework successfully retroactively explained why: the explanations offered almost zero human-complementary information because humans are already highly proficient at sentiment analysis.
  • Mechanistic Interpretability Alignment Audit: The theoretic value plateaued after showing the top 3 Sparse Autoencoder (SAE) features. However, simulated human performance kept increasing up to 5 features, revealing a gap where agents under-extracted available information, motivating the need for better user training.

Limitations and Future Work

  • Coarsening Dependency: The framework relies heavily on the coarsening algorithm to handle high-dimensional data. If the clustering is suboptimal, the estimated benchmarks might be loose.
  • Simulated Humans: For the alignment audit, the authors used a Large Language Model (Qwen3-14B) to simulate human decisions. While useful for scaling, LLMs may not perfectly mirror human cognitive biases or constraints.
  • Static Policies: The framework assumes a static baseline human policy, whereas real humans might adapt, learn, or grow fatigued when interacting with explanations over time.

Summary updated

Abstract

Explanations of model behavior are commonly evaluated via proxy properties weakly tied to the purposes explanations serve in practice. We contribute a decision theoretic framework that treats explanations as information signals valued by the expected improvement they enable on a specified decision task. This approach yields three distinct estimands: 1) a theoretical benchmark that upperbounds achievable performance by any agent with the explanation, 2) a human-complementary value that quantifies the theoretically attainable value that is not already captured by a baseline human decision policy, and 3) a behavioral value representing the causal effect of providing the explanation to human decision-makers. We instantiate these definitions in a practical validation workflow, and apply them to assess explanation potential and interpret behavioral effects in human-AI decision support and mechanistic interpretability.
BibTeX
@inproceedings{guo2026explanations,
  author = {Ziyang Guo and Berk Ustun and Jessica Hullman},
  title = {{Explanations are a Means to an End: Decision Theoretic Explanation Evaluation}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=fVso2kGVrF},
}
Figure