CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models

Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, Wenhao Zheng, Zhaoyang Wang, Xiao Wang, Xuchao Zhang, Chetan Bansal, Marc Niethammer, Junzhou Huang, Hongtu Zhu, Yun Li, Jimeng Sun, Zongyuan Ge, Gang Li, James Zou, Huaxiu Yao

TL;DR

The deployment of Medical Large Vision Language Models (Med-LVLMs) in healthcare is bottlenecked by unverified reliability. To address this, the authors introduce CARES, a comprehensive benchmark that evaluates Med-LVLMs across five core dimensions of trustworthiness: trustfulness, fairness, safety, privacy, and robustness. Testing reveals that current models suffer from severe factual hallucinations, demographic biases, and critical vulnerabilities to adversarial exploits.

What Problem Does This Paper Address?

Artificial intelligence has the potential to revolutionize medicine, with Med-LVLMs promising automated diagnostics and personalized treatment plans. However, an unreliable model in a clinical setting can have catastrophic consequences—such as misidentifying a benign tumor as malignant, leading to unnecessary invasive surgeries and severe patient distress.

While previous research has evaluated these models, evaluations typically focus narrowly on diagnostic accuracy. A standardized, multi-dimensional framework to assess broader trustworthiness—including safety, patient privacy, and demographic fairness—has been entirely missing. This paper bridges that gap by establishing a rigorous evaluation protocol to uncover the hidden risks of deploying Med-LVLMs.

Figure 1: CARES is designed to provide a comprehensive evaluation of trustworthiness in Med- LVLMs, reflecting the issues present in model responses. We assess trustworthiness across five critical dimensions: trustfulness, fairness, safety, privacy, and robustness.

Methods

The authors developed a standardized evaluation pipeline across five critical dimensions:

  1. Trustfulness: Assessed via factuality (using GPT-4 to score open-ended responses) and uncertainty estimation (prompting models with "are you sure?" to evaluate overconfidence).
  2. Fairness: Measuring performance disparities across different age, gender, and racial groups using demographic accuracy differences.
  3. Safety: Testing vulnerability to jailbreaking prompts (e.g., inducing the model to exaggerate conditions), toxicity generation, and overcautiousness (unnecessary refusal to answer routine queries).
  4. Privacy: Evaluating zero-shot and few-shot leakage of Protected Health Information (PHI), specifically marital status and Social Security Numbers.
  5. Robustness: Testing out-of-distribution (OOD) performance by adding Gaussian noise to images and shifting semantic modalities.

The benchmark evaluates four specialized open-source medical models (LLaVA-Med, Med-Flamingo, MedVInT, RadFM) alongside two state-of-the-art generic models (Qwen-VL-Chat and LLaVA-v1.6) for comparison.

Datasets

The CARES benchmark curates approximately 41K question-answer pairs and 18K images from seven open-source medical multimodal and image classification datasets: MIMIC-CXR, IU-Xray, Harvard-FairVLMed, PMC-OA, HAM10000, OL3I, and OmniMedVQA. The benchmark spans 16 medical image modalities (including X-ray, MRI, CT, and pathology) and 27 distinct anatomical regions.

Statistical overview of the anatomical structures and imaging modalities included in CARES.

Key Insights and Ideas

  • The Illusion of Certainty: Med-LVLMs are highly prone to medical hallucinations but express them with extreme overconfidence, making them uniquely dangerous for clinical decision support.
  • Data Imbalance Drives Demographic Bias: Models perform best on patients aged 40–60 but show a sharp decline in accuracy for elderly populations, directly reflecting the skewed age distribution in public medical training data.
  • The Safety-Utility Trade-off: LLaVA-Med demonstrated the strongest resistance to jailbreaking and toxic prompts, but this came at the cost of extreme overcautiousness, resulting in a 90% refusal rate when asked for routine medical advice.
  • Privacy Blindness: Unlike general-purpose models that are heavily aligned to protect user data, medical models lack robust privacy guardrails, frequently leaking or completely fabricating sensitive patient PHI when prompted.

Key Results

  • Low Factuality Across the Board: Every single model evaluated experienced significant hallucinations, with average factual accuracy scores falling below 50%. LLaVA-Med achieved the highest average accuracy at 40.39%, followed closely by MedVInT at 39.31%, while RadFM scored the lowest at 27.51%.
  • Poor Uncertainty Calibration: Uncertainty-based accuracy was largely below 50%, proving that models cannot reliably identify the limits of their own medical knowledge.
  • Vulnerability to Noise: Introducing Gaussian noise to input images caused sharp drops in diagnostic accuracy, yet the models rarely exercised their option to abstain or refuse to answer.
Figure 3: Accuracy (%) on factuality evalua- tion. Above are the performance comparisons of all models across 7 datasets, and below are the average performances of each model. “Mixture” represents mixtures of modalities.

Limitations and Future Work

The authors note that despite CARES' broad coverage, limitations in open-source data availability prevent the benchmark from covering every possible medical modality and anatomical structure. Furthermore, because the pretraining datasets of many commercial and open-source models are undisclosed, absolute protection against evaluation data leakage cannot be guaranteed.

Moving forward, the authors advocate for incorporating manual clinical expert assessments to complement automated VQA metrics, expanding the benchmark to include broader ethical considerations, and evaluating closed-source frontier models like Med-Gemini.

Summary updated

Abstract

Artificial intelligence has significantly impacted medical applications, particularly with the advent of Medical Large Vision Language Models (Med-LVLMs), sparking optimism for the future of automated and personalized healthcare. However, the trustworthiness of Med-LVLMs remains unverified, posing significant risks for future model deployment. In this paper, we introduce CARES and aim to comprehensively evaluate the Trustworthiness of Med-LVLMs across the medical domain. We assess the trustworthiness of Med-LVLMs across five dimensions, including trustfulness, fairness, safety, privacy, and robustness. CARES comprises about 41K question-answer pairs in both closed and open-ended formats, covering 16 medical image modalities and 27 anatomical regions. Our analysis reveals that the models consistently exhibit concerns regarding trustworthiness, often displaying factual inaccuracies and failing to maintain fairness across different demographic groups. Furthermore, they are vulnerable to attacks and demonstrate a lack of privacy awareness. We publicly release our benchmark and code in https://cares-ai.github.io/.
BibTeX
@inproceedings{xia2024cares,
  author = {Peng Xia and Ze Chen and Juanxi Tian and Yangrui Gong and Ruibo Hou and Yue Xu and Zhenbang Wu and Zhiyuan Fan and Yiyang Zhou and Kangyu Zhu and Wenhao Zheng and Zhaoyang Wang and Xiao Wang and Xuchao Zhang and Chetan Bansal and Marc Niethammer and Junzhou Huang and Hongtu Zhu and Yun Li and Jimeng Sun and Zongyuan Ge and Gang Li and James Zou and Huaxiu Yao},
  title = {{CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure