Abstract
Medical data mining is traditionally hindered by the unstructured nature of clinical reports and the high variability of medical imaging. This study benchmarks state-of-the-art Multimodal Large Language Models (MLLMs) across 14 diverse datasets to evaluate their potential as generalist tools for medical image analysis and report generation. The results demonstrate a performance split: Gemini-series models excel in text-heavy report generation and lesion detection, while GPT-series models show superior capabilities in anatomical localization and complex image segmentation. Despite their promise, significant hurdles regarding domain-specific accuracy and processing efficiency remain.
Introduction
Medical data is notoriously difficult to mine. Radiology reports are often written in unstructured "free-text," and medical images—including X-rays, fundus photos, and endoscopies—vary significantly in quality and appearance across different clinical settings. Historically, AI models in medicine have been "specialists," trained for specific tasks on specific organs. This research investigates whether the new wave of "Generalist" Multimodal Large Language Models (MLLMs), such as GPT-4o and Gemini 1.5, can act as a versatile "Swiss Army Knife" for medicine, handling diverse tasks like diagnosing pneumonia or summarizing chest reports without task-specific fine-tuning.
Methodology
The researchers conducted a comprehensive zero-shot evaluation, testing the models' "out-of-the-box" knowledge without additional fine-tuning. The study compared 10 prominent models, including OpenAI’s GPT-4o and GPT-4-Turbo, Google’s Gemini-1.5-Pro and Flash, and other leading models like Claude-3-Opus and Llama 3.
The evaluation focused on three primary pillars:
- Medical Image Question Answering (QA): Using standardized prompts to assess the models' ability to diagnose, localize, or segment findings within images.
- Medical Report Generation: Testing the models' ability to synthesize clinical conclusions from findings, measured by the ROUGE metric (a standard for text similarity).
- Efficiency: Measuring "Character Per Time (ms)" to determine which models are viable for real-world clinical workflows.
Data Sources
The study utilized a massive battery of 14 datasets to ensure a broad clinical scope:
- Imaging (11 datasets): Included iChallenge GON (glaucoma), MICCAI Tooth (dentistry), ChestXRay2017 and COVID-QU (radiology), CholecSeg8k and Kvasir SEG (endoscopy), and the ISIC/HAM10000 series (dermatology).
- Reports (3 datasets): Included the widely used MIMIC-CXR, the public OpenI dataset, and a private dataset (SXY) from Xiangya Second Hospital.
Comparative Insights
The Specialization Split
Despite their generalist design, the models exhibited distinct "personalities." Gemini models proved to be superior "writers," producing more accurate and structured radiology reports. In contrast, GPT models were more effective "observers," demonstrating a significantly higher capability for identifying the exact spatial location of lesions within an image.
The "Zero-Shot" Wall
All models struggled significantly with dermatological classification (skin cancer). This suggests that for highly nuanced visual tasks, general knowledge is insufficient; these models still require domain-specific fine-tuning or more sophisticated prompting strategies to achieve clinical reliability.
Granularity in Specialized Imaging
In dental imaging tasks, Gemini models tended to provide simple tooth counts. However, GPT-4 and Claude-3-Opus provided much more granular assessments, identifying bone structures, implants, and wisdom teeth, and even offering preliminary clinical recommendations.
The Localization Challenge
Localization remains a significant weakness for current MLLMs. Most models failed to provide precise coordinates for anatomical findings (such as the macular fovea in eye scans), often resorting to vague descriptions or providing incorrect spatial data.
Key Results and Efficiency
- Report Generation: Gemini-1.0-Pro-Vision achieved the highest ROUGE-1 score (0.2814) on the MIMIC-CXR dataset, notably outperforming GPT-4o (0.2275).
- Diagnostic Accuracy: GPT-series models were the most accurate in identifying pneumonia from chest X-rays without requiring supplementary prompts.
- Processing Speed: GPT-4o was the fastest model across almost all categories, making it the most viable candidate for real-time clinical use.
- The Efficiency Gap: Gemini-1.5-Pro was the slowest model tested, taking roughly 9.16 times longer than GPT-4o to generate the same volume of text. This latency could be a significant barrier in high-pressure medical environments.
Limitations and Future Directions
Several hurdles must be cleared before these models can be integrated into clinical practice:
- Catastrophic Forgetting: Fine-tuning a model for one specialty (e.g., X-rays) often degrades its performance in another (e.g., dermatology). Maintaining "generalist" capabilities during specialization is a key research priority.
- Hallucinations: Models occasionally "see" non-existent findings or miss obvious lesions, which is unacceptable in high-stakes medical environments.
- Validation Metrics: The study calls for more rigorous clinical trials and the development of metrics that measure clinical accuracy rather than just text overlap (ROUGE).
Conclusion
MLLMs show immense promise as "medical interns" capable of reducing administrative and diagnostic workloads. However, they are not yet ready to function as independent "attending physicians." Future work must focus on grounding these models in clinical reality to ensure patient safety and diagnostic precision.
Summary updated