TL;DR
The deployment of Medical Large Vision Language Models (Med-LVLMs) in healthcare is bottlenecked by unverified reliability. To address this, the authors introduce CARES, a comprehensive benchmark that evaluates Med-LVLMs across five core dimensions of trustworthiness: trustfulness, fairness, safety, privacy, and robustness. Testing reveals that current models suffer from severe factual hallucinations, demographic biases, and critical vulnerabilities to adversarial exploits.
What Problem Does This Paper Address?
Artificial intelligence has the potential to revolutionize medicine, with Med-LVLMs promising automated diagnostics and personalized treatment plans. However, an unreliable model in a clinical setting can have catastrophic consequences—such as misidentifying a benign tumor as malignant, leading to unnecessary invasive surgeries and severe patient distress.
While previous research has evaluated these models, evaluations typically focus narrowly on diagnostic accuracy. A standardized, multi-dimensional framework to assess broader trustworthiness—including safety, patient privacy, and demographic fairness—has been entirely missing. This paper bridges that gap by establishing a rigorous evaluation protocol to uncover the hidden risks of deploying Med-LVLMs.
Methods
The authors developed a standardized evaluation pipeline across five critical dimensions:
- Trustfulness: Assessed via factuality (using GPT-4 to score open-ended responses) and uncertainty estimation (prompting models with "are you sure?" to evaluate overconfidence).
- Fairness: Measuring performance disparities across different age, gender, and racial groups using demographic accuracy differences.
- Safety: Testing vulnerability to jailbreaking prompts (e.g., inducing the model to exaggerate conditions), toxicity generation, and overcautiousness (unnecessary refusal to answer routine queries).
- Privacy: Evaluating zero-shot and few-shot leakage of Protected Health Information (PHI), specifically marital status and Social Security Numbers.
- Robustness: Testing out-of-distribution (OOD) performance by adding Gaussian noise to images and shifting semantic modalities.
The benchmark evaluates four specialized open-source medical models (LLaVA-Med, Med-Flamingo, MedVInT, RadFM) alongside two state-of-the-art generic models (Qwen-VL-Chat and LLaVA-v1.6) for comparison.
Datasets
The CARES benchmark curates approximately 41K question-answer pairs and 18K images from seven open-source medical multimodal and image classification datasets: MIMIC-CXR, IU-Xray, Harvard-FairVLMed, PMC-OA, HAM10000, OL3I, and OmniMedVQA. The benchmark spans 16 medical image modalities (including X-ray, MRI, CT, and pathology) and 27 distinct anatomical regions.
Key Insights and Ideas
- The Illusion of Certainty: Med-LVLMs are highly prone to medical hallucinations but express them with extreme overconfidence, making them uniquely dangerous for clinical decision support.
- Data Imbalance Drives Demographic Bias: Models perform best on patients aged 40–60 but show a sharp decline in accuracy for elderly populations, directly reflecting the skewed age distribution in public medical training data.
- The Safety-Utility Trade-off: LLaVA-Med demonstrated the strongest resistance to jailbreaking and toxic prompts, but this came at the cost of extreme overcautiousness, resulting in a 90% refusal rate when asked for routine medical advice.
- Privacy Blindness: Unlike general-purpose models that are heavily aligned to protect user data, medical models lack robust privacy guardrails, frequently leaking or completely fabricating sensitive patient PHI when prompted.
Key Results
- Low Factuality Across the Board: Every single model evaluated experienced significant hallucinations, with average factual accuracy scores falling below 50%. LLaVA-Med achieved the highest average accuracy at 40.39%, followed closely by MedVInT at 39.31%, while RadFM scored the lowest at 27.51%.
- Poor Uncertainty Calibration: Uncertainty-based accuracy was largely below 50%, proving that models cannot reliably identify the limits of their own medical knowledge.
- Vulnerability to Noise: Introducing Gaussian noise to input images caused sharp drops in diagnostic accuracy, yet the models rarely exercised their option to abstain or refuse to answer.
Limitations and Future Work
The authors note that despite CARES' broad coverage, limitations in open-source data availability prevent the benchmark from covering every possible medical modality and anatomical structure. Furthermore, because the pretraining datasets of many commercial and open-source models are undisclosed, absolute protection against evaluation data leakage cannot be guaranteed.
Moving forward, the authors advocate for incorporating manual clinical expert assessments to complement automated VQA metrics, expanding the benchmark to include broader ethical considerations, and evaluating closed-source frontier models like Med-Gemini.
Summary updated