Abstract
Recently emerging large multimodal models (LMMs) utilize various data modalities, including text and visual inputs, to generate outputs. The incorporation of LMMs into clinical medicine presents unique challenges, including accuracy, reliability, and clinical relevance. This study explores the clinical applications of GPT-4V in gastroenterology, radiology, dermatology, and United States Medical Licensing Examination (USMLE) test questions. Using standardized datasets of endoscopy images, chest X-rays, and skin lesions, we benchmarked GPT-4V’s ability to predict diagnoses and assessed potential bias regarding Fitzpatrick skin tones. We found that GPT-4V is limited in performance across all four domains, resulting in decreased performance compared to previously published specialized baseline models. For instance, the macro-average F1-score for gastroenterology was only 6.8%. In radiology, identifying cardiomegaly yielded a precision of 28% and a sensitivity of 94%. In dermatology, GPT-4V had a top-1 diagnostic accuracy of 6.2%, with a significant accuracy drop when predicting images of darker skin tones (p < 0.001). Our findings demonstrate that the current version of GPT-4V is limited in its diagnostic abilities across multiple image-based medical specialties, suggesting a need for hybrid models or improved prompting strategies.
Introduction
The medical field is inherently multimodal, relying on a combination of patient history (text) and diagnostic imaging (visuals). While Large Language Models (LLMs) have excelled at text-based medical exams, the integration of vision into these models—creating Large Multimodal Models (LMMs)—represents a new frontier. This research addresses a critical gap: the lack of rigorous, multi-domain benchmarking for general-purpose LMMs in clinical settings. Before these models can assist clinicians, it is vital to determine if a "jack-of-all-trades" model like GPT-4V can match the accuracy of specialized medical AI or human specialists, and whether it carries inherent biases that could harm specific patient populations.
Methodology
Experimental Design
The researchers conducted a comprehensive evaluation of GPT-4V (as of October 2023) using a zero-shot prompting strategy. The model was asked to perform tasks without specific medical fine-tuning or prior examples. Prompt engineering was employed to assign the model a persona (e.g., "You are an AI doctor specializing in Radiology") and provide a list of possible diagnoses to minimize hallucinations.
Datasets and Metrics
The study utilized four robust, standardized datasets:
- Gastrovision: 1,587 endoscopy images covering 27 distinct classes of gastrointestinal findings.
- CheXpert: 668 chest X-rays used to identify five clinically relevant diseases (e.g., Atelectasis, Cardiomegaly).
- Diverse Dermatology Images (DDI): 656 clinical images categorized by the Fitzpatrick skin tone (FST) scale.
- Visual USMLE: 618 image-based questions adapted from the medical licensing examination.
Performance was measured against established "State-of-the-Art" (SOTA) specialized models and, in dermatology, against an ensemble of board-certified dermatologists. Metrics included the Matthews Correlation Coefficient (MCC), F1-scores, and p-values to assess diagnostic accuracy and demographic disparities.
Results
Gastroenterology: The "Generalist" Penalty
The most striking finding was the significant performance gap between general-purpose LMMs and specialized models. GPT-4V lacks the specialized feature extraction capabilities of models trained specifically on medical data, leading to a massive disparity in diagnostic accuracy.
As shown in Table 1, GPT-4V achieved a macro F1-score of only 6.8%, compared to 65% for the specialized Gastrovision DenseNet-121 model.
Radiology and Dermatology
In radiology, GPT-4V was excellent at "seeing" certain conditions but poor at ruling them out. For identifying cardiomegaly, it achieved a high sensitivity (94%) but very low precision (28%), making it a poor fit for a primary diagnostic tool.
In dermatology, the model's top-1 diagnostic accuracy was a meager 6.2%. When compared to human experts on malignancy detection, dermatologists achieved 67.99% accuracy versus GPT-4V’s 39.62%. GPT-4V showed a high propensity to over-predict malignant conditions, which, while cautious, leads to extremely low precision and potential over-testing.
Algorithmic Bias and Skin Tone Prediction
The study confirmed that general LMMs inherit the biases of their training data. There was a statistically significant drop in top-3 accuracy for darker skin tones (FST V-VI, p < 0.001). Furthermore, the model struggled to even correctly identify the skin tone group of the images it was analyzing.
As illustrated in Figure 1, the model frequently misclassified skin tone groups, which likely contributes to the observed diagnostic performance disparities across different patient demographics.
Discussion and Limitations
Key Insights
- Specialization vs. Generalization: General LMMs currently struggle with the nuances of medical imaging compared to specialized encoders.
- Baseline Knowledge: On the Visual USMLE, GPT-4V scored 59%, showing a baseline of medical knowledge that nonetheless falls short of expert-level performance.
- Equitable Healthcare: The "diversity gap" in AI training remains a major hurdle, as evidenced by the performance drop on darker skin tones.
Limitations and Future Work
The authors acknowledge that results might vary with more advanced prompting techniques, such as "Chain-of-Thought" or "few-shot" prompting. Additionally, as a closed-source model, the exact training data for GPT-4V remains unknown. Future research should explore hybrid models that combine the reasoning power of LMMs with the precision of specialized medical vision encoders to ensure clinical safety and accuracy.
Summary updated