Position: Explainable AI Cannot Advance Without Better User Studies

ICML 2025

TL;DR

This paper argues that user studies are crucial for understanding the real-world value of Explainable AI (XAI), but current XAI user studies are often of poor quality, lacking reproducibility and real-world fidelity. The authors call for higher methodological standards and a greater appreciation for robust user studies to truly advance the field.

What Problem Does This Paper Address?

Explainable AI (XAI) aims to make AI systems more understandable, trustworthy, and fair. While there's a growing academic interest in XAI, the authors argue that its ultimate goal is to satisfy societal desiderata like trust and fairness, which can only be truly assessed through human interaction – hence, user studies. However, they contend that the current landscape of XAI user studies is problematic. Many studies are poorly designed, lack rigor, and don't accurately reflect how XAI will be used in real-world applications. This deficiency hinders our understanding of XAI's actual value and impedes the field's progress.

Methods

The paper's core contribution is a position paper supported by an empirical analysis of 607 XAI papers featuring user studies. The authors conducted a systematic literature search across multiple databases and top AI/ML conferences to identify relevant papers. They then analyzed these papers based on various criteria, including publication year, participant count and type, and methodological quality. This analysis involved categorizing studies based on evaluation type (e.g., subjective satisfaction, forward simulatability) and fidelity (e.g., high, medium, low, no application context).

Datasets

Not discussed in the paper. The paper focuses on analyzing existing XAI research papers rather than introducing new datasets or benchmarks.

Key Insights and Ideas

  • User studies are non-negotiable for XAI's true value: The authors firmly believe that understanding the societal impact of XAI (trust, fairness, etc.) necessitates human evaluation. Functional evaluation alone is insufficient.
  • The current state of XAI user studies is "poor": This is a central theme. Many studies suffer from a lack of reproducibility, insufficient discussion of limitations, absence of baselines, and low fidelity to real-world scenarios.
  • "Placebo effect" is a significant concern: Just like in medicine, explanations might influence user perception or trust even if they are not truly informative. Many XAI studies don't account for this, making their results questionable.
  • Crowdsourcing platforms (like MTurk) are prevalent but problematic: While convenient, these platforms often yield data from participants who may be less attentive, more prone to satisficing, and potentially less representative of the general population.
  • A disconnect exists between academic XAI and real-world needs: Many studies use simplified tasks or toy applications, failing to capture the complexity of real-world decision-making scenarios.
  • The field needs to embrace negative results: The authors echo the sentiment that negative findings are valuable and should be published, as they can reveal limitations and guide future research more effectively than overly optimistic results.

Key Results

The empirical analysis revealed several key findings:

  • Publication trends: XAI research with user studies has grown rapidly, with a significant increase in papers from 2020 onwards.
  • Participant demographics: Crowdsourcing platforms (especially MTurk) and student recruitment are the most common methods for obtaining participants, often making up over two-thirds of studies. Domain experts and AI experts are less frequently included.
  • Methodological quality:
    • Reproducibility is a major issue, with estimates suggesting only 50-90% of papers are reproducible, and even fewer in the top XAI venues.
  • Validated questionnaires, preregistration, and pretesting are rare.
  • Baselines are included in about half of the studies, but placebo-controlled studies are very rare.
  • Discussions of limitations are present in about half of the studies.
  • Evaluation fidelity: Most studies evaluate XAI in "toy" applications or without any application context ("No" fidelity). Only a small fraction of studies achieve high fidelity, embedding XAI in real-world applications.
  • Evaluation types: Subjective satisfaction and subjective comparison are the most common evaluation types.

Limitations and Future Work

The authors acknowledge several limitations:

  • Search bias: Their sample is biased towards certain venues and publication years due to reliance on Scopus and keyword-based searches. They also note that their search might have missed papers that don't use typical XAI or user study keywords.
  • Rater variability: While they used a single rater for key variables, there's a possibility of between-rater variability in subjective assessments.
  • Limited scope: The study is exploratory, and a more systematic confirmatory study with a broader range of venues (including human-centric research venues) is needed.

They suggest future work should focus on:

  • Developing standardized benchmarks: Linking functional evaluation metrics, proxy tasks, and real-world performance is crucial.
  • Improving generalizability: Understanding how to generalize findings from crowdsourced and student-based studies to broader populations.
  • Investigating user differences: Exploring intra-user and inter-user differences in how people interact with and perceive XAI.
  • Promoting placebo-controlled studies: Developing and encouraging the use of placebo-controlled designs to better isolate the true effect of explanations.
  • Conducting neutral method comparisons: Directly comparing popular XAI methods to get a clearer picture of their effectiveness.
  • Elevating the quality of XAI user studies: Encouraging better design, reporting, and review practices within the ML community, potentially through dedicated venues or tracks.

Summary updated

BibTeX
@inproceedings{2025position,
  title = {{Position: Explainable AI Cannot Advance Without Better User Studies}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
}
Figure