CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models

Peng Xia, Ze Chen, Juanxi Tian, Gong Yangrui, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, Wenhao Zheng, Zhaoyang Wang, Xiao Wang, Xuchao Zhang, Chetan Bansal, Marc Niethammer, Junzhou Huang, Hongtu Zhu, Yun Li, Jimeng Sun, Zongyuan Ge, Gang Li, James Zou, Huaxiu Yao

TL;DR

This paper introduces CARES, a comprehensive benchmark for evaluating the trustworthiness of Medical Large Vision Language Models (Med-LVLMs). Their analysis reveals that current Med-LVLMs exhibit significant issues with factual accuracy, fairness, safety, and privacy, highlighting the urgent need for more reliable models in healthcare.

What Problem Does This Paper Address?

The rapid advancement of AI in healthcare, particularly with Medical Large Vision Language Models (Med-LVLMs), promises to revolutionize patient care. These models can analyze medical images and provide textual insights, aiding in diagnosis and treatment. However, a critical concern is their trustworthiness. If Med-LVLMs provide inaccurate information, exhibit biases, or mishandle sensitive patient data, the consequences could be severe, leading to misdiagnoses, health disparities, and privacy breaches. Existing evaluations often focus on single aspects of trustworthiness, leaving a gap in a holistic understanding of their reliability for real-world medical applications.

Methods

The authors developed CARES, a comprehensive benchmark designed to evaluate Med-LVLMs across five key dimensions:

  • Trustfulness: Assesses factual accuracy (factuality) and the model's ability to express uncertainty when unsure.
  • Fairness: Examines performance disparities across demographic groups (age, gender, race).
  • Safety: Evaluates resistance to "jailbreaking" (manipulative prompts), overcautiousness, and toxicity in responses.
  • Privacy: Tests the models' ability to protect sensitive patient information.
  • Robustness: Assesses performance under out-of-distribution (OOD) conditions, including noisy images and unfamiliar modalities.

CARES utilizes a combination of existing medical vision-language and image classification datasets. They constructed approximately 41,000 question-answer pairs, encompassing 16 medical image modalities and 27 anatomical regions, in both closed-ended (multiple-choice, yes/no) and open-ended formats. GPT-4 was used to help generate and evaluate open-ended question-answer pairs.

Datasets

CARES is built upon data from seven sources:

  • Medical Vision-Language Datasets: MIMIC-CXR, IU-Xray, Harvard-FairVLMed, PMC-OA.
  • Medical Image Classification Datasets: HAM10000, OL3I.
  • Large-Scale VQA Dataset: OmniMedVQA.

These datasets cover a wide range of modalities (X-ray, MRI, CT, Pathology, Fundus, Dermoscopy, etc.) and anatomical regions (chest, lung, eye, skin, etc.). The benchmark itself comprises around 18,000 images and 41,000 QA pairs.

Key Insights and Ideas

  • The "Trustworthiness Gap": The most striking insight is that current Med-LVLMs, despite their impressive capabilities, are far from trustworthy for clinical deployment. They consistently fall short across multiple critical dimensions.
  • Factual Inaccuracies are Rampant: Models frequently "hallucinate" information, providing incorrect diagnoses or descriptions, especially with rare modalities or open-ended questions. This is a major red flag for patient safety.
  • Fairness is a Significant Concern: Performance disparities exist across age, gender, and race, with models often performing worse for the elderly and certain racial groups. This could exacerbate existing health inequities.
  • Vulnerability to Attacks: Med-LVLMs show poor resistance to jailbreaking prompts and can be induced to produce toxic or incorrect advice, posing security risks.
  • Privacy is Largely Ignored: Models demonstrate a significant lack of awareness regarding patient privacy, with some even leaking information from input prompts.
  • Overcautiousness vs. Robustness Trade-off: Some models, like LLaVA-Med, exhibit strong safety measures by refusing to answer, but this can lead to overcautiousness, hindering usability even for routine queries. Conversely, many models are not robust to noisy data or OOD inputs.

Key Results

  • Factuality: Most models achieved less than 50% accuracy on the factuality evaluation.
  • Fairness: Age-related disparities were notable, with models performing best in the 40-60 age group and declining for the elderly. Racial disparities were also observed, with better performance for Caucasian/Hispanic populations.
  • Safety: All models showed reduced accuracy under jailbreaking attacks. LLaVA-Med demonstrated better resistance by abstaining from some attacks, but also exhibited severe overcautiousness.
  • Privacy: Models showed significant shortfalls in privacy protection, with low abstention rates on privacy-related questions.
  • Robustness: Models exhibited poor OOD robustness, with low abstention rates even when presented with noisy or out-of-distribution data, indicating a tendency to guess rather than refuse.

Limitations and Future Work

The authors acknowledge several limitations:

  • Data Coverage: While CARES covers many modalities and regions, limitations in available open-source medical data mean not all areas are fully represented.
  • Data Leakage: Despite efforts to use only test sets, there's a possibility that some data might have been included in the pre-training corpora of the evaluated LLMs.
  • Evaluation Scope: While CARES covers five key dimensions, other aspects of trustworthiness, such as ethical considerations, could be explored further.

Future work suggested includes:

  • Clinical Expert Assessment: Incorporating human expert evaluations for more nuanced and accurate assessments.
  • Expanded Evaluation Dimensions: Exploring additional trustworthiness aspects like ethical considerations.
  • Richer Data Sources: As more medical data becomes available, expanding the benchmark to cover more modalities and anatomical sites.
  • More SOTA Models: Continuously evaluating newer and more advanced Med-LVLMs as they emerge.

Summary updated

Abstract

Artificial intelligence has significantly impacted medical applications, particularly with the advent of Medical Large Vision Language Models (Med-LVLMs), sparking optimism for the future of automated and personalized healthcare. However, the trustworthiness of Med-LVLMs remains unverified, posing significant risks for future model deployment. In this paper, we introduce CARES and aim to comprehensively evaluate the Trustworthiness of Med-LVLMs across the medical domain. We assess the trustworthiness of Med-LVLMs across five dimensions, including trustfulness, fairness, safety, privacy, and robustness. CARES comprises about 41K question-answer pairs in both closed and open-ended formats, covering 16 medical image modalities and 27 anatomical regions. Our analysis reveals that the models consistently exhibit concerns regarding trustworthiness, often displaying factual inaccuracies and failing to maintain fairness across different demographic groups. Furthermore, they are vulnerable to attacks and demonstrate a lack of privacy awareness. We publicly release our benchmark and code in https://github.com/richard-peng-xia/CARES.
BibTeX
@inproceedings{xia2024cares,
  author = {Peng Xia and Ze Chen and Juanxi Tian and Gong Yangrui and Ruibo Hou and Yue Xu and Zhenbang Wu and Zhiyuan Fan and Yiyang Zhou and Kangyu Zhu and Wenhao Zheng and Zhaoyang Wang and Xiao Wang and Xuchao Zhang and Chetan Bansal and Marc Niethammer and Junzhou Huang and Hongtu Zhu and Yun Li and Jimeng Sun and Zongyuan Ge and Gang Li and James Zou and Huaxiu Yao},
  title = {{CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure