Abstract
Our results demonstrate the susceptibility of radiology readers to automation bias in detecting cerebral aneurysms in TOF-MRA studies when encountering false-positive AI findings. While AI systems for cerebral aneurysm detection can provide benefits, challenges in human-AI interaction need to be mitigated to ensure safe and effective adoption.
Introduction
Artificial Intelligence (AI) is increasingly integrated into clinical workflows as a "second reader" to assist radiologists in identifying cerebral aneurysms. These vascular bulges, though present in only about 2% of the population, carry a high risk of fatal subarachnoid hemorrhage if they rupture. However, the low prevalence of aneurysms introduces a "false-positive paradox": even highly accurate AI systems can generate a high volume of false alarms relative to true detections.
This study explores the phenomenon of automation bias—the tendency for human operators to over-trust automated systems and stop critically evaluating their output. Specifically, the researchers investigated whether false-positive AI markers lead radiologists to misidentify normal vascular structures as suspicious, potentially triggering unnecessary and invasive follow-up procedures.
Methods and Study Design
The researchers conducted a controlled reader study involving nine radiologists categorized by experience:
- Inexperienced: Residents with less than one year of neuroradiology experience.
- Moderately Experienced: Board-certified radiologists.
- Very Experienced: Certified neuroradiologists.
The participants evaluated 20 Time-of-Flight Magnetic Resonance Angiography (TOF-MRA) cases across two sessions, separated by a four-week washout period to minimize memory effects. In one session, cases were read conventionally; in the other, readers were assisted by a commercially available AI CAD system (mdbrain, version 4; Mediaire GmbH).
The dataset was intentionally "salted" with 10 true-positive cases and 10 cases containing at least one false-positive AI finding (e.g., vascular loops or normal junctions misidentified by the software). Readers rated findings on a 4-point Likert scale and recommended follow-up actions ranging from "None" to "Invasive Angiography."
Key Findings
The Impact of Experience
The study revealed that clinical experience serves as a significant safeguard against automation bias. While very experienced neuroradiologists remained largely unaffected by false-positive AI markers, junior residents were highly susceptible. When presented with an AI-flagged false positive, inexperienced readers' suspicion levels rose significantly ($p=0.002$), often leading them to recommend unnecessary follow-up MRIs.
Sensitivity vs. Specificity
AI assistance successfully boosted overall sensitivity from 0.88 to 0.97 ($p=0.02$). However, this gain came at a cost. For inexperienced readers, the presence of AI markers led to a significant increase in follow-up intensity for non-pathological findings ($p=0.005$).
The Efficiency Paradox
AI significantly reduced reading times across all groups, with average times dropping from 179.4 to 136.0 seconds. For the inexperienced group, the reduction was most dramatic (228.2s to 164.1s). However, this increased speed likely contributed to automation bias, as readers may have used AI markers as shortcuts rather than conducting a thorough independent review.
Discussion and Conclusion
Qualitative feedback highlighted a "medico-legal shadow": some experienced readers noted they might recommend follow-up for an AI-flagged finding they personally doubted, simply to mitigate the legal risk of overriding a machine-generated warning.
While AI offers clear benefits in sensitivity and workflow efficiency, this study underscores the risks of over-reliance, particularly among trainees. Future implementation of AI in radiology must include robust human-AI interaction protocols to ensure that "second readers" do not inadvertently become "only readers." Further research is needed to quantify the economic impact of AI-induced unnecessary follow-ups and to refine training for radiologists working alongside automated systems.
Summary updated