Abstract
Vision-language models (VLMs) can answer clinically relevant questions with their reasoning capabilities and user-friendly interfaces. However, their robustness to commonly existing medical image artefacts has not been explored, leaving major concerns in trustworthy clinical decision-making. In this study, we assessed the robustness of recent VLMs to medical image artefacts in disease detection across three different medical fields. Specifically, we included five categories of image artefacts, and evaluated the VLMs’ performance on images with and without artefacts. We build evaluation benchmarks in brain MRI, Chest X-ray, and retinal images, involving four real-world medical datasets. Our results demonstrate that VLMs showed poor performance on original unaltered images and performed even worse when weak artefacts were introduced. The strong artefacts were barely detected by those VLMs. Our findings indicate that VLMs are not yet capable of performing medical tasks with image artefacts, underscoring the critical need to explicitly incorporate artefact-aware method design and robustness tests into VLM development.
Introduction
The integration of Vision-Language Models (VLMs) into clinical workflows promises to revolutionize diagnostic assistance through intuitive, natural-language interactions. However, the transition from curated research datasets to the "wild" environment of a hospital introduces a significant challenge: image artefacts. In practice, medical imaging is frequently compromised by patient movement, hardware limitations, or suboptimal acquisition techniques. While traditional deep learning models have been rigorously tested against such distortions, the robustness of the latest generation of VLMs—including proprietary models like GPT-4o and open-source alternatives—remains largely unquantified.
This study addresses a fundamental safety gap: Can VLMs maintain diagnostic integrity when faced with imperfect data? If a model misinterprets motion blur as a lesion or fails to recognize that an image is ungradable, it poses a direct risk to patient safety.
Methodology
The researchers established a comprehensive robustness evaluation framework spanning three critical medical modalities: Brain MRI, Chest X-ray, and Retinal Optical Coherence Tomography (OCT).
Artefact Injection and Categorization
To simulate real-world conditions, the study introduced five types of artefacts into clean images, categorized into two groups:
- Intensity-based Artefacts: These include random noise, motion blur, and bias fields, which alter the pixel values and clarity of the image.
- Spatial-based Artefacts: These include cropping and rotation, which alter the orientation or completeness of the visual field.
These distortions were applied at two severity levels: "weak" (where features are partially obscured but potentially interpretable) and "strong" (where the image is rendered mostly ungradable by human standards).
Model Selection and Prompting Strategies
The evaluation included a diverse set of models:
- Proprietary Models: GPT-4o and Claude 3.5 Sonnet.
- Open-Source Models: Llama 3.2 11B and the domain-specific BiomedCLIP.
To determine if the method of interaction influenced robustness, three prompting strategies were employed: Structured (forcing a binary choice), Standard (open-ended), and Chain-of-Thought (CoT) (encouraging step-by-step reasoning).
Datasets
The study utilized four primary real-world datasets to ensure broad applicability:
- Brain MRI: A combination of Figshare, SARTAJ, and Br35H datasets.
- Retinal OCT: The Kermany 2018 dataset.
- Chest X-ray: The COVID-19 Detection X-ray Dataset.
- Color Fundus (Real-world): The Diabetic Retinopathy (DDR) dataset, specifically chosen for its "ungradable" category to validate synthetic findings.
Key Insights
Intensity vs. Spatial Sensitivity
A primary finding was that VLMs are significantly more vulnerable to intensity-based artefacts than spatial ones. Random noise, in particular, caused the most dramatic performance degradation. Conversely, spatial artefacts like cropping occasionally improved accuracy, likely by focusing the model's attention on specific regions or reducing the "distraction" of healthy tissue surrounding a pathology.
The "Blind Spot" for Quality Assessment
Perhaps the most concerning discovery was the models' inability to identify ungradable images. When presented with "strong" artefacts that would lead a human clinician to reject the scan, models often attempted a diagnosis regardless. Claude 3.5 Sonnet demonstrated the highest level of "self-awareness" by flagging poor quality in some instances, whereas GPT-4o frequently opted to refuse the prompt entirely rather than identifying the specific quality issue.
The Chain-of-Thought Paradox
While CoT prompting is known to improve logical reasoning in Large Language Models (LLMs), its impact on VLM robustness was inconsistent. For GPT-4o, CoT often triggered a refusal to answer rather than a more resilient diagnosis. This suggests that while reasoning might help the model identify uncertainty, it does not necessarily improve the underlying visual perception required to overcome image degradation.
Domain-Specific Performance
BiomedCLIP, despite being smaller than general-purpose giants like GPT-4o, often achieved superior performance in MRI and X-ray tasks. This underscores the value of pre-training on medical-specific image-text pairs, which likely provides the model with a more robust internal representation of clinical features.
Results and Discussion
The baseline performance on clean images was surprisingly modest, with accuracy peaking at approximately 0.77. The introduction of even "weak" random noise led to a "noise tax," where sensitivity dropped sharply—for instance, BiomedCLIP’s MRI sensitivity fell from 0.95 to 0.78.
Furthermore, the detection rate for "strong" artefacts was highly erratic. While Claude 3.5 Sonnet could detect noise in MRI with 94.5% accuracy, it failed completely (0.0% detection) to recognize strong rotation as a quality issue. Meanwhile, Llama 3.2 11B exhibited a tendency toward over-diagnosis, frequently classifying images as "diseased" regardless of quality, leading to high sensitivity but unacceptably low overall accuracy.
Conclusion and Future Directions
The study concludes that current VLMs are not yet ready for autonomous clinical use in environments where image quality cannot be guaranteed. The "brittleness" of these models necessitates several shifts in development:
- Quality-First Architecture: VLMs should incorporate a dedicated quality assessment module that evaluates image usability before attempting a diagnosis.
- Robustness Training: Future models must be explicitly trained on "noisy" data to improve their resilience to common clinical artefacts.
- Fine-Grained Evaluation: Moving beyond binary "normal vs. diseased" tasks to more complex multi-class classifications will better reflect clinical reality.
Ultimately, while VLMs offer powerful reasoning capabilities, their visual perception remains a critical bottleneck that must be addressed to ensure patient safety in AI-assisted medicine.
Summary updated