Who should do as AI say? Only non-task expert physicians benefit from correct explainable AI advice

Susanne Gaube, Harini Suresh, Martina Raue, Eva Lermer, T. Koch, Matthias F. C. Hudecek, Alun Ackery, Samir C. Grover, Joseph F. Coughlin, Dieter Frey, Felipe Kitamura, Marzyeh Ghassemi, Errol ColakResearch Square 2022 · DOI

TL;DR

While AI models often match or beat human experts in medical imaging, their actual utility depends heavily on who is using them. This study finds that explainable AI advice (using visual annotations) significantly boosts the diagnostic accuracy of non-specialist physicians, but provides little benefit to specialists, who may already be performing at their "ceiling."

What Problem Does This Paper Address?

We are currently in a "gold rush" of AI-enabled clinical decision support systems (AI-CDSS), with hundreds of radiology products receiving regulatory clearance. However, a high-performing algorithm doesn't automatically lead to better patient outcomes. The "human-in-the-loop" remains the final gatekeeper.

The authors address a critical gap: How do the source of advice (AI vs. Human) and the explainability of that advice (visual cues vs. just a diagnosis) affect doctors with different levels of expertise? Specifically, they look at whether a radiologist (a task expert) and an internal medicine or emergency physician (a non-task expert) interact with AI advice differently.

Methods

The researchers conducted an online experiment with 223 physicians from the US and Canada. The cohort was split into "task experts" (106 radiologists) and "non-task experts" (117 IM/EM physicians).

Each participant reviewed eight chest X-ray cases and received diagnostic advice that was always correct. The study used a 2x2 factorial design, manipulating two variables:

  1. Explainability: The advice either came with a visual annotation (arrows pointing to the pathology) or without one.
  2. Source Label: The advice was labeled as coming from either "CHEST-AI" (a deep-learning model) or "Dr. S. Johnson" (an experienced radiologist).

Participants then provided their own diagnosis, rated the quality of the advice, and reported their confidence levels. The researchers used mixed-effects regression models to analyze the impact on diagnostic accuracy, quality ratings, and confidence.

Datasets

  • MIMIC-CXR: The study utilized frontal and lateral chest X-rays sourced from this large, open-source database.
  • Expert-Curated Cases: A panel of three radiologists selected eight cases representing everyday clinical practice and common diagnostic pitfalls. One senior radiologist added patient vignettes and verified the "ground truth" annotations.

Key Insights and Ideas

  • The "Expertise Ceiling": Explainability (visual annotations) was a game-changer for non-experts but did nothing for radiologists. This suggests that experts already have the internal "mental map" to find the pathology; they don't need the AI to point it out. Non-experts, however, lack this specialized training and rely on the AI's "explanation" to validate the advice.
  • Algorithmic Appreciation: In a surprising twist, physicians performed better when they thought the advice came from an AI rather than a human colleague. This contradicts some previous findings of "algorithmic aversion" (where people distrust machines). Here, the AI label actually boosted accuracy and, for experts, increased confidence.
  • Closing the Gap: Perhaps the most striking insight is that non-experts receiving annotated AI advice performed almost as well as experts who received no annotations. This suggests AI could be a powerful "equalizer" in settings where specialists aren't available, such as rural clinics or night shifts in the ER.

Key Results

  • Accuracy Boost: Non-experts saw a significant increase in diagnostic accuracy when advice was annotated. For experts, the effect was negligible.
  • The Power of the AI Label: Across the board, participants had higher odds of a correct diagnosis when the advice was labeled as "AI" (Odds Ratio = 2.09) compared to "Human."
  • Subjective Quality: Non-experts rated annotated advice as higher quality than non-annotated advice. Interestingly, the perceived source (AI vs. Human) did not change the quality rating, even though it changed the doctors' actual performance.
  • Confidence: Radiologists reported significantly higher confidence in their final diagnosis when they believed they were being supported by an AI.

Limitations and Future Work

The most significant limitation, which the authors openly acknowledge, is that the study only provided correct advice. In the real world, AI models fail. A major concern in AI safety is "automation bias"—where a user blindly follows a machine's incorrect suggestion. Future research needs to test if explainability (like those helpful arrows) makes it harder for a doctor to spot when the AI is wrong.

Additionally, the study was a low-stakes online simulation. In a high-pressure clinical environment, the way a doctor weighs AI advice against their own intuition might change. Finally, the small number of cases (six analyzed) limits the ability to generalize these findings across all types of radiological pathologies.

Summary updated

Abstract

<title>Abstract</title> Artificial intelligence (AI)-generated clinical advice is becoming more prevalent in healthcare. However, the impact of AI-generated advice on physicians’ decision-making is underexplored. In this study, physicians received X-rays with correct diagnostic advice and were asked to make a diagnosis, rate the advice’s quality, and judge their own confidence. We manipulated whether the advice came with or without a visual annotation on the X-rays and whether it was labeled as coming from an AI or a human radiologist. Overall, receiving annotated advice from an AI resulted in the highest diagnostic accuracy. Physicians rated the quality of AI advice higher than human advice. Neither manipulation had strong effects on participants’ confidence. Importantly, the results varied among task experts and non-task experts, with only the latter considerably benefiting from correct explainable AI advice. These findings raise important considerations for the deployment of diagnostic advice in healthcare.
BibTeX
@inproceedings{gaube2022who,
  author = {Susanne Gaube and Harini Suresh and Martina Raue and Eva Lermer and T. Koch and Matthias F. C. Hudecek and Alun Ackery and Samir C. Grover and Joseph F. Coughlin and Dieter Frey and Felipe Kitamura and Marzyeh Ghassemi and Errol Colak},
  title = {{Who should do as AI say? Only non-task expert physicians benefit from correct explainable AI advice}},
  booktitle = {Research Square},
  year = {2022},
  doi = {10.21203/rs.3.rs-1687219/v1},
}
Figure