The Human Evaluation Gap in Explainable AI

37 concepts29 papers1 proposal

Explainable Artificial Intelligence (XAI) is built on a fundamental promise: making complex, "black-box" models understandable to the people who use them. However, a growing body of research suggests the field suffers from an "original sin"—a systemic disconnect between the technical metrics used to build explanations and the actual cognitive needs of human beings.

The 1% Problem

The most striking evidence of this gap is the sheer lack of empirical testing. In a systematic analysis of over 18,000 papers, Suh et al. found that less than 1% of research claiming to provide "explainable" methods actually conducts empirical tests with human subjects.

This lack of validation means that most XAI methods are optimized for mathematical proxies rather than human utility. Researchers often assume that if an explanation is mathematically sound—for example, using game-theoretic indices like the Shapley value—it will naturally align with human trust and decision-making. Yet, as Labreuche et al. note, these formal measures remain largely unverified in complex, real-world domains.

Mathematical Sparsity vs. Human Intuition

A core tension in XAI research is the definition of a "good" explanation. Many algorithms prioritize sparsity (using as few features as possible) and proximity (making the smallest possible change to an input). The intuition is that simpler is better.

However, Delaney et al. demonstrate that this technical intuition is often wrong. In studies of counterfactual explanations—which tell a user what would need to change to get a different outcome—they found that humans actually prefer large, semantic edits that move an image toward a "prototype" of a new class, rather than the minimal pixel nudges favored by algorithms.

This suggests that by optimizing for technical metrics like sparsity, researchers may be creating explanations that satisfy an optimizer but fail to help a human process information (Delaney et al.). As Gjoreski et al. argue, mathematical excellence does not guarantee that an explanation is helpful or intuitive for the individuals it is intended to assist.

The Risk of Misplaced Trust

The evaluation gap isn't just a missed opportunity for better design; it poses a safety risk. Without rigorous human-centric testing, it is impossible to know if an explanation fosters appropriate reliance or dangerous overreliance.

Visser et al. clarify that the goal of XAI should not just be to increase trust, but to help users know when not to trust a model. If an explanation looks convincing but doesn't actually reveal the model's errors, it may lead users to follow flawed AI advice blindly. This is particularly critical in high-stakes environments like healthcare or aviation, where Suh et al. argue that XAI lacks the rigorous "clinical trials" necessary to ensure safe human-AI collaboration.

Bridging the Gap

To move the field forward, researchers are calling for a shift from formal logic puzzles to real-world impact. This involves several key changes:

  • Psychologically Grounded Frameworks: Moving beyond model-derived explanations to consider how AI outputs interact with existing human intuition (Chen et al.).
  • Actionable Sequences: Evaluating whether explanations provide a clear path for user compliance, such as the ordered sequences proposed by Kanamori et al..
  • Domain-Specific Validation: Testing explanations with the actual experts who will use them—such as doctors or business analysts—rather than relying on small convenience samples of non-experts (Suh et al.; Tonati et al.).

As Rago et al. argue, the transition from theoretical frameworks to production-ready software requires proving that these complex reasoning models actually enhance human-AI collaboration through rigorous, empirical user studies. Without this evidence, the "explainability" of AI remains a technical assumption rather than a human reality.

Key Foundational Work

Integrated modelling of business process models and business rules: a research agenda (2014) — contributed 2 concept elements to this direction.
How Do You Tell a Blackbird from a Crow? (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_user_centered_evaluation_gap_0aa6f08d,
  title = {user-centered evaluation gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/user-centered-evaluation-gap/rdr_0aa6f08d5217413f90e234c7c0b20464},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_user_centered_evaluation_gap_0aa6f08d_v1,
  title = {user-centered evaluation gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/user-centered-evaluation-gap/rdr_0aa6f08d5217413f90e234c7c0b20464/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake