Underreliance Harms Human-AI Collaboration More Than Overreliance in Medical Imaging

Susanne Gaube, Ekaterina Jussupow, Eesha Kokje, Jamal Nasir Khan, Elizabeth Bondi, Andreas Schicho, Felipe Kitamura, T. Koch, Timur Ezer, Jürgen Mottok, Eva Lermer, Marzyeh Ghassemi, Errol Colak

Abstract

Importance: The use of artificial intelligence (AI) to support clinicians in diagnostic decision-making holds significant potential; however, evidence regarding its clinical utility remains mixed. In many cases, the interaction between healthcare professionals and AI systems does not improve collaborative performance compared to the standalone performance of humans or AI. Currently, the underlying mechanisms that limit human-AI collaboration are poorly understood.Objective: To examine the impact of AI advice on diagnostic decision-making among experts and novices, focusing on understanding the role of explainability (XAI) on users’ reliance on advice.Design, Setting, and Participants: A mixed-methods design combining a crossover experimental design with a think-aloud and an eye-tracking study arm was conducted in 2023. Participants were task experts (radiologists) and novices (non-radiologist physicians and medical trainees) from 10 countries, with the think-aloud and eye-tracking conducted in Germany.Intervention: Participants reviewed 50 patient cases containing head CT scans and patient information. Every case was reviewed in three time-separate sessions in randomized order. In each session, participants were exposed to a different experimental condition: (a) control, i.e., no AI prediction presented; (b) basic advice, i.e., AI prediction without annotations; and (c) XAI advice, i.e., AI prediction with annotations. For each case, participants had to determine if the patients had an intracranial hemorrhage (ICH), rate their confidence, and, if applicable, the usefulness of the AI advice.Main Outcome(s) and Measure(s): Diagnostic performance, confidence in the diagnosis, case reading time, and AI advice usefulness ratings.Results: The data analysis included 125 participants. The mean age was 28.5 years (SD = 6.72), and 55.2% identified as female. Underreliance on correct AI advice was associated with high uncertainty and had a more detrimental impact on diagnostic performance than overreliance on incorrect advice. XAI advice reduced underreliance and improved performance and confidence, particularly when reviewing more difficult cases with ICH. AI advice, particularly XAI, did not reduce reading time. XAI was perceived as more useful than basic AI advice, especially among novices.Conclusions and Relevance: Our findings indicate that underreliance on AI might be more harmful than overreliance, highlighting the need to develop efficient counterstrategies beyond current XAI methods.
BibTeX
@misc{gaube2024underreliance,
  author = {Susanne Gaube and Ekaterina Jussupow and Eesha Kokje and Jamal Nasir Khan and Elizabeth Bondi and Andreas Schicho and Felipe Kitamura and T. Koch and Timur Ezer and Jürgen Mottok and Eva Lermer and Marzyeh Ghassemi and Errol Colak},
  title = {{Underreliance Harms Human-AI Collaboration More Than Overreliance in Medical Imaging}},
  year = {2024},
  doi = {10.31219/osf.io/4wv8j},
}

Loading related content...

Figure