Abstract
OpenAI’s large multimodal model, GPT-4V(ision), was recently developed for general image interpretation. However, less is known about its capabilities with medical image interpretation and diagnosis. Board-certified physicians and senior residents assessed GPT-4V’s proficiency across a range of medical conditions using imaging modalities such as CT scans, MRIs, ECGs, and clinical photographs. Although GPT-4V is able to identify and explain medical images, its diagnostic accuracy and clinical decision-making abilities are poor, posing risks to patient safety. Despite the potential that large language models may have in enhancing medical education and delivery, the current limitations of GPT-4V in interpreting medical images reinforces the importance of appropriate caution when using it for clinical decision-making.
Introduction
The advent of multimodal large language models (LLMs) like GPT-4V has generated significant interest in their potential application within specialized fields such as diagnostic radiology, cardiology, and dermatology. While LLMs have demonstrated the ability to pass medical licensing exams using text-based inputs, their capacity to accurately "see" and interpret complex medical imagery remains a critical frontier. This study evaluates whether GPT-4V’s descriptive capabilities translate into safe and accurate clinical reasoning, addressing the risk of misdiagnosis and the dissemination of harmful medical misinformation.
Methods
Researchers utilized a "human-in-the-loop" framework to evaluate GPT-4V across a diverse dataset of 69 multimodal medical images. These included CT scans (head, chest, abdomen), MRIs, chest X-rays, 24 ECG waveforms, and 49 clinical dermatology photographs.
The model was prompted as an "expert tutor" to describe the major abnormalities in each image. The resulting outputs were then rigorously reviewed by a panel of experts, including board-certified physicians and senior surgical residents. Evaluators graded the model on diagnostic accuracy, the safety of the advice provided, and the overall utility of the content for medical education.
Results and Key Insights
The evaluation revealed a significant discrepancy between the model's professional-sounding prose and its actual diagnostic accuracy—a phenomenon termed the "Eloquence Trap."
Diagnostic Inaccuracy and the "Eloquence Trap"
GPT-4V often correctly identified the type of imaging modality but failed to detect overt, life-threatening pathologies. For instance, the model incorrectly identified a clear pneumothorax as a "benign nodule." Overall, only 15 out of 69 multimodal images were interpreted correctly.
High Risk of Clinical Harm
A primary concern identified was the frequency of "dangerous" advice. Clinicians flagged 43% of multimodal interpretations and 37.5% of ECG interpretations as containing advice that could jeopardize patient safety. In cardiology specifically, GPT-4V failed to match the performance of standard automated ECG software, struggling with fundamental metrics like heart rate and axis deviation.
Educational Unsuitability
The expert panel expressed strong reservations about using GPT-4V as a teaching tool. The average comfort level for medical students relying on this content was only 1.8 out of 5. The risk of reinforcing incorrect diagnostic patterns outweighed the model's ability to provide layman-friendly descriptions.
Discussion and Limitations
While GPT-4V shows promise in general image description, its current state is unsuitable for clinical care. Several limitations were noted, including the model's lack of access to patient history and physical exam data, which are essential for real-world diagnosis. Furthermore, despite the potential for data leakage from public repositories, the model's performance remained poor.
Future developments may require moving away from general-purpose models toward domain-specific, fine-tuned architectures trained on curated medical datasets and aligned with stringent safety standards. Until then, extreme caution is warranted when considering multimodal LLMs for clinical decision-making.
Summary updated