Fewer Than 1% of Explainable AI Papers Validate Explainability with Humans

Ashley Suh, Isabelle Hurley, Nora Smith, Ho Chit Siu

TL;DR

A massive literature review of over 18,000 Explainable AI (XAI) papers reveals a shocking truth: fewer than 1% (0.7%) actually test their methods with human users. Despite claiming to make AI transparent and understandable, the vast majority of XAI research relies purely on algorithmic metrics or anecdotal evidence. This critical gap highlights an urgent need for empirical human validation in a field explicitly designed for human benefit.

What Problem Does This Paper Address?

Explainable AI (XAI) aims to open the "black box" of machine learning, making models trustworthy and interpretable for human end-users like doctors, pilots, or financial analysts. However, there is a growing suspicion that XAI researchers are building tools for humans without actually testing them on humans. While previous small-scale surveys hinted at this issue, this paper provides a comprehensive, large-scale audit of the XAI literature to quantify exactly how widespread this lack of empirical validation really is.

Methods

The authors teamed up with a professional librarian to design a rigorous search strategy across the Scopus database, capturing a decade of XAI literature including both peer-reviewed papers and preprints (from platforms like arXiv).

They filtered the initial massive pool of papers down to a subset containing keywords related to human participation (e.g., "human feedback", "user interview", "end-user evaluation"). Three researchers independently reviewed and scored the final subset using a binary metric adapted from prior literature: a score of 1 if the paper conducted an empirical human study to validate the XAI method, and 0 otherwise. Reasons for a 0 included papers claiming to be "explainable-by-design" without testing, or using human-annotated data for training but not for evaluation.

Datasets

Instead of standard machine learning benchmarks, the "dataset" here is the broader body of academic literature. The authors scraped 18,254 XAI-related papers (13,839 published papers and 4,394 preprints) from the Scopus database.

Key Insights and Ideas

  • The "Explainability" Illusion: The XAI community is largely talking to itself. By treating explainability as an algorithmic property rather than a human psychological phenomenon, researchers are essentially "sending signals into a void" without checking if anyone receives or understands them.
  • Misleading Keywords: Even when papers explicitly use terms like "user evaluation" or "human feedback" in their text, nearly half (46%) do not actually perform a behavioral study with humans.
  • Low-Quality Evaluations: Among the tiny fraction (0.7%) that did conduct human studies, the quality was often poor. Many relied on small sample sizes of college students rather than target domain experts (like clinicians or security analysts) or used superficial binary questions like "Is this explainable: Yes or No?".

Key Results

  • Out of 18,254 total XAI papers, only 253 (1.39%) contained keywords suggesting human involvement.
  • After removing off-topic papers (like meta-reviews), 237 papers were manually evaluated.
  • Only 128 papers successfully provided empirical evidence from a human study. This represents a mere 0.70% of the entire XAI literature over the last decade.

The dramatic drop-off from the total literature to validated papers is illustrated in the authors' down-selection distribution:

Figure 1: Distribution comparing all XAI literature from our Scopus search (Table 1), including those we scored. ‘All XAI papers’ is the superset that encompasses all papers with keywords related to explainability, interpretability, etc. ‘Claims about humans’ papers are a subset of those that were filtered based on keywords related to human explainability. ‘On topic’ papers are a subset of those that we filtered to exclude meta reviews, surveys, etc. ‘Validated’ papers are a subset of those that provided empirical evidence of explainability. On the right a zoomed in version of the last three bars from the left figure are shown, note the change in x-axes.

Furthermore, of the papers that did validate, a significant portion featured very small participant pools, often under 15 subjects, raising questions about generalizability:

Figure 2: Distribution of counts for human subjects involved in an evaluation, experiment, interview, etc. for validated XAI papers. It is important to note that 13 papers did not report their human subject count and 3 papers approximated their count. In the bottom figure, a subset of distribution counts up to N=152 and a bin size of 5.

Limitations and Future Work

The authors acknowledge that their search criteria excluded workshop papers, review articles, and non-English publications, which might introduce some bias. Additionally, some niche XAI papers focus strictly on scientific discovery (e.g., using explanations to uncover physical mechanisms) where broad human-interpretability testing might genuinely be unnecessary.

Future work involves refining the search criteria to better categorize different types of XAI research, reworking older evaluation frameworks to match modern standards, and analyzing whether human-validated papers are more frequently published in human-computer interaction (HCI) venues versus mainstream AI/ML conferences.

Summary updated

Abstract

This late-breaking work presents a large-scale analysis of explainable AI (XAI) literature to evaluate claims of human explainability. We collaborated with a professional librarian to identify 18,254 papers containing keywords related to explainability and interpretability. Of these, we find that only 253 papers included terms suggesting human involvement in evaluating an XAI technique, and just 128 of those conducted some form of a human study. In other words, fewer than 1% of XAI papers (0.7%) provide empirical evidence of human explainability when compared to the broader body of XAI literature. Our findings underscore a critical gap between claims of human explainability and evidence-based validation, raising concerns about the rigor of XAI research. We call for increased emphasis on human evaluations in XAI studies and provide our literature search methodology to enable both reproducibility and further investigation into this widespread issue.
BibTeX
@inproceedings{suh2025fewer,
  author = {Ashley Suh and Isabelle Hurley and Nora Smith and Ho Chit Siu},
  title = {{Fewer Than 1\% of Explainable AI Papers Validate Explainability with Humans}},
  booktitle = {CHI Extended Abstracts},
  year = {2025},
  doi = {10.1145/3706599.3719964},
}
Figure