TL;DR
A massive literature review of over 18,000 Explainable AI (XAI) papers reveals a shocking truth: fewer than 1% (0.7%) actually test their methods with human users. Despite claiming to make AI transparent and understandable, the vast majority of XAI research relies purely on algorithmic metrics or anecdotal evidence. This critical gap highlights an urgent need for empirical human validation in a field explicitly designed for human benefit.
What Problem Does This Paper Address?
Explainable AI (XAI) aims to open the "black box" of machine learning, making models trustworthy and interpretable for human end-users like doctors, pilots, or financial analysts. However, there is a growing suspicion that XAI researchers are building tools for humans without actually testing them on humans. While previous small-scale surveys hinted at this issue, this paper provides a comprehensive, large-scale audit of the XAI literature to quantify exactly how widespread this lack of empirical validation really is.
Methods
The authors teamed up with a professional librarian to design a rigorous search strategy across the Scopus database, capturing a decade of XAI literature including both peer-reviewed papers and preprints (from platforms like arXiv).
They filtered the initial massive pool of papers down to a subset containing keywords related to human participation (e.g., "human feedback", "user interview", "end-user evaluation"). Three researchers independently reviewed and scored the final subset using a binary metric adapted from prior literature: a score of 1 if the paper conducted an empirical human study to validate the XAI method, and 0 otherwise. Reasons for a 0 included papers claiming to be "explainable-by-design" without testing, or using human-annotated data for training but not for evaluation.
Datasets
Instead of standard machine learning benchmarks, the "dataset" here is the broader body of academic literature. The authors scraped 18,254 XAI-related papers (13,839 published papers and 4,394 preprints) from the Scopus database.
Key Insights and Ideas
- The "Explainability" Illusion: The XAI community is largely talking to itself. By treating explainability as an algorithmic property rather than a human psychological phenomenon, researchers are essentially "sending signals into a void" without checking if anyone receives or understands them.
- Misleading Keywords: Even when papers explicitly use terms like "user evaluation" or "human feedback" in their text, nearly half (46%) do not actually perform a behavioral study with humans.
- Low-Quality Evaluations: Among the tiny fraction (0.7%) that did conduct human studies, the quality was often poor. Many relied on small sample sizes of college students rather than target domain experts (like clinicians or security analysts) or used superficial binary questions like "Is this explainable: Yes or No?".
Key Results
- Out of 18,254 total XAI papers, only 253 (1.39%) contained keywords suggesting human involvement.
- After removing off-topic papers (like meta-reviews), 237 papers were manually evaluated.
- Only 128 papers successfully provided empirical evidence from a human study. This represents a mere 0.70% of the entire XAI literature over the last decade.
The dramatic drop-off from the total literature to validated papers is illustrated in the authors' down-selection distribution:
Furthermore, of the papers that did validate, a significant portion featured very small participant pools, often under 15 subjects, raising questions about generalizability:
Limitations and Future Work
The authors acknowledge that their search criteria excluded workshop papers, review articles, and non-English publications, which might introduce some bias. Additionally, some niche XAI papers focus strictly on scientific discovery (e.g., using explanations to uncover physical mechanisms) where broad human-interpretability testing might genuinely be unnecessary.
Future work involves refining the search criteria to better categorize different types of XAI research, reworking older evaluation frameworks to match modern standards, and analyzing whether human-validated papers are more frequently published in human-computer interaction (HCI) venues versus mainstream AI/ML conferences.
Summary updated