Abstract
This late-breaking work presents a large-scale analysis of explainable AI (XAI) literature to evaluate claims of human explainability. We collaborated with a professional librarian to identify 18,254 papers containing keywords related to explainability and interpretability. Of these, we find that only 253 papers included terms suggesting human involvement in evaluating an XAI technique, and just 128 of those conducted some form of a human study. In other words, fewer than 1% of XAI papers (0.7%) provide empirical evidence of human explainability when compared to the broader body of XAI literature. Our findings underscore a critical gap between claims of human explainability and evidence-based validation, raising concerns about the rigor of XAI research. We call for increased emphasis on human evaluations in XAI studies and provide our literature search methodology to enable both reproducibility and further investigation into this widespread issue.
Introduction
Explainable AI (XAI) is built on the premise of making complex AI models understandable to humans. As artificial intelligence is increasingly deployed in high-stakes domains like healthcare, autonomous driving, and disaster forecasting, the need for transparency has skyrocketed. Developers of XAI methods argue that their techniques make "black box" models trustworthy and usable for human end-users. However, a troubling trend has emerged: XAI researchers frequently claim their methods are "explainable" based purely on algorithmic metrics, proxy heuristics (such as rule conciseness), or the authors' own intuition. This research investigates a critical gap: out of all XAI papers claiming human explainability, how many actually validate those claims with empirical evidence from real humans?
Methodology
To quantify the gap between XAI claims and empirical human testing, a large-scale literature review was conducted using the Scopus database. The process involved three main stages:
- Broad Search: In collaboration with a professional librarian, the authors queried Scopus for English-language XAI papers and preprints published over the last decade. Keywords included "explainable," "interpretable," and "transparent" in conjunction with AI/ML terms. This yielded a corpus of 18,254 papers.
- Keyword Filtering: This corpus was filtered for papers explicitly including terms suggesting human testing, such as "human-subject," "end-user," "interview," "feedback," or "evaluation."
- Manual Scoring: A team of three reviewers manually evaluated the resulting 253-paper subset. Papers were scored as
1if they conducted a human experiment to test the proposed XAI method and0otherwise (e.g., if they only used human-annotated data for training or claimed "explainability-by-design" without testing).
Key Results
The quantitative findings reveal a stark disconnect between the field's objectives and its validation practices. Of the 18,254 total XAI papers identified, only 253 (1.39%) contained keywords suggesting human evaluation. Upon manual review, only 128 papers (0.70% of the total) actually provided empirical evidence of human explainability.
Statistically, for every 1,000 XAI papers published, fewer than 8 provide empirical evidence that their explanations actually work for humans.
Discussion: The Void of Validation
The lack of human testing suggests that the XAI field is largely "sending signals into a void." An explanation is a communication signal; without confirming that the signal was received and understood, the claim of explainability remains unverified.
The "Intuitively Obvious" Fallacy
Many researchers skip human testing by assuming their methods are inherently understandable—for instance, because they use natural language or short rule sets. However, brevity does not automatically equate to human comprehension. Furthermore, nearly half (46%) of the papers that passed the initial human-keyword filter did not actually contain empirical human testing, highlighting a "keyword mirage" in the literature.
Methodological Weakness
Even within the 0.7% of papers that conducted human studies, rigor was often lacking. Many relied on convenience samples (e.g., college students) rather than domain experts like doctors or pilots. Furthermore, sample sizes were generally very small.
As shown in the distribution above, the majority of studies involved very few participants, with 13 papers failing to report their subject count entirely. The authors liken this state of affairs to a pharmaceutical company releasing a drug based on simulations without conducting clinical trials.
Conclusion and Future Work
This study highlights a critical need for the XAI community to treat human evaluation as the gold standard for explainability claims. While the current search was limited to English-language papers, the findings suggest a systemic issue. Future work will involve expanding the search to include non-English papers and workshop proceedings to mitigate bias, alongside a qualitative deep dive into the 128 validated papers to identify best practices for XAI evaluation frameworks.
Summary updated