When evaluating an explainable AI (XAI) system, it is tempting to rely entirely on mathematical proxies. We often assume that if an explanation method is axiomatically sound, highly sparse, or faithful to the underlying model's internal weights, it must be useful to a human operator.
However, a growing body of research reveals a profound interpretability-utility gap: an explanation can be technically flawless according to algorithmic metrics while failing completely to help humans understand, trust, or debug a system. To bridge this gap, researchers are shifting toward human-centric validation—evaluating explanations not by their mathematical properties, but by their measurable impact on human performance.
The Failure of Technical Proxies
Why can't we trust automated metrics to tell us if an explanation is good? The core issue is that mathematical optimization does not inherently map to human cognition.
Consider a feature attribution map that highlights the pixels or words driving a model's prediction. Vivian Lai et al. demonstrated in Many Faces of Feature Importance: Comparing Built-in and Post-hoc Feature Importance in Text Classification that characterizing the statistical properties of these explanations (such as entropy) fails to indicate their practical utility or whether they foster appropriate trust. Similarly, Gagan Biradar et al. argued in Axiomatic Aggregations of Abductive Explanations that even when an explanation method satisfies strict mathematical axioms, it does not guarantee human interpretability or real-world utility.
This disconnect manifests in several ways:
- The Clarity Illusion: Technical adjustments that improve the visual clarity or faithfulness of an attribution map do not automatically translate into a better human ability to debug a model, as shown by Sukrut Rao et al. in Better Understanding Differences in Attribution Methods via Systematic Evaluations.
- Metric Gaming: Automated metrics can often be optimized or "cheated" by dummy explanations that align with mathematical criteria but offer zero conceptual value to an end-user, a vulnerability highlighted by Marcella Cornia et al. in Revisiting the Evaluation of Class Activation Mapping for Explainability: A Novel Metric and Experimental Analysis.
Ultimately, if an explanation is meant to inform a human, its validation must happen through the lens of human capability.
Operationalizing Human Centric Evaluation
To move past proxy metrics, researchers design empirical user studies that treat the human-AI pair as a unified system. Instead of asking users if they "like" an explanation, these protocols measure changes in objective performance metrics across specific tasks.
Simulatability (Forward Prediction)
One of the most rigorous tests of an explanation is whether it allows a human to mentally simulate the model. In a forward-prediction task, a user is shown an input and its corresponding explanation, and is asked to guess what the model will predict.
Dong Nguyen 0002 utilized this approach in Comparing Automatic and Human Evaluation of Local Explanations for Text Classification, uncovering a significant disconnect: automatic metrics did not reliably predict whether an explanation actually helped humans anticipate model behavior. This concept of simulatability is vital for dynamic systems; for instance, Xuelin Situ et al. emphasized in Lifelong Explainer for Lifelong Learners that evaluating how effectively an explanation allows a person to predict a model's evolving behavior is a truer test of interpretability than static technical metrics.
Causability-Usability Assessment
An explanation works when a user can map the provided information to a functional, cause-and-effect mental model of the machine's logic. Oliver Hinz et al. framed this in Challenges in Interactive Machine Learning as a "causability-usability" assessment. Rather than evaluating transparency in a vacuum, this metric quantifies the exact extent to which an explanation improves a user's decision-making performance and understanding of system causality.
Defect Detection and Debugging
Can an explanation help a human catch a broken model? In deployment scenarios, we need to know if explanations expose hidden flaws, such as a model relying on spurious background correlations. Julius Adebayo et al. established a testing protocol in Debugging Tests for Model Explanations that explicitly measures whether feature attributions successfully enable human subjects to detect hidden model defects, tracking whether users rely blindly on output labels or accurately interpret the explanation signals.
[Model + Input] ──> [Explanation Method] ──> [Human User] ──> Measure:
├── Simulatability (Can they predict the output?)
├── Causability (Do they understand the cause?)
└── Debugging (Can they spot hidden defects?)
Cognitive Misalignments: Minimality vs. Plausibility
Human-centric validation frequently reveals that what is mathematically optimal is often cognitively jarring. A prime example of this occurs in counterfactual explanations, which show how an input would need to change to flip a model's prediction (e.g., "If your income were \$10,000 higher, your loan would be approved").
Algorithmic approaches typically optimize for minimality—finding the smallest possible mathematical edit to change the output class, often resulting in highly sparse modifications, as explored by André Artelt et al. in "Why Here and Not There?" - Diverse Contrasting Explanations of Dimensionality Reduction.
However, when evaluated with real users, this assumption falls apart. Eoin D. Delaney et al. demonstrated in Counterfactual Explanations for Misclassified Images: How Human and Machine Explanations Differ (Abstract Reprint) that humans do not want minimal, pixel-level changes. Instead, they prefer larger, "meaningful" edits that align with typical, recognizable examples of a different category.
To address this, human-in-the-loop evaluations are shifting toward measuring plausibility. In Generating and Evaluating Plausible Explanations for Knowledge Graph Completion, Zhao Xu 0001 showed that prioritizing human-centric plausibility over raw algorithmic metrics directly improves the accuracy of a user's assessment of a prediction, while simultaneously reducing the time required to reach a decision. Zhao Xu 0001 also noted that standard benchmarks are often poorly suited for this validation because they lack semantic transparency, highlighting the need for datasets curated specifically for human understandability.
The Double-Edged Sword of XAI
Perhaps the most compelling argument for human-centric validation is that explanations are not a universal good; they can sometimes actively hinder human performance.
In a clinical deployment study, Angus Nicolson et al. evaluated the real-world utility of XAI in The human factor in explainable artificial intelligence: clinician variability in trust, reliance, and performance. The authors discovered that while AI assistance generally reduced errors, the addition of visual explanations introduced highly variable performance among clinicians, and in some cases, actually hindered expert decision-making.
This finding underscores why technical validation alone is insufficient. An explanation method can be perfectly faithful to a model, but if it induces cognitive overload, creates false confidence, or distracts an expert, its practical utility is negative. True validation requires observing the human factor in collaborative settings to ensure that transparency leads to better decisions, rather than misplaced trust.