TL;DR
This paper introduces CARES, a comprehensive benchmark for evaluating the trustworthiness of Medical Large Vision Language Models (Med-LVLMs). Their analysis reveals that current Med-LVLMs exhibit significant issues with factual accuracy, fairness, safety, and privacy, highlighting the urgent need for more reliable models in healthcare.
What Problem Does This Paper Address?
The rapid advancement of AI in healthcare, particularly with Medical Large Vision Language Models (Med-LVLMs), promises to revolutionize patient care. These models can analyze medical images and provide textual insights, aiding in diagnosis and treatment. However, a critical concern is their trustworthiness. If Med-LVLMs provide inaccurate information, exhibit biases, or mishandle sensitive patient data, the consequences could be severe, leading to misdiagnoses, health disparities, and privacy breaches. Existing evaluations often focus on single aspects of trustworthiness, leaving a gap in a holistic understanding of their reliability for real-world medical applications.
Methods
The authors developed CARES, a comprehensive benchmark designed to evaluate Med-LVLMs across five key dimensions:
- Trustfulness: Assesses factual accuracy (factuality) and the model's ability to express uncertainty when unsure.
- Fairness: Examines performance disparities across demographic groups (age, gender, race).
- Safety: Evaluates resistance to "jailbreaking" (manipulative prompts), overcautiousness, and toxicity in responses.
- Privacy: Tests the models' ability to protect sensitive patient information.
- Robustness: Assesses performance under out-of-distribution (OOD) conditions, including noisy images and unfamiliar modalities.
CARES utilizes a combination of existing medical vision-language and image classification datasets. They constructed approximately 41,000 question-answer pairs, encompassing 16 medical image modalities and 27 anatomical regions, in both closed-ended (multiple-choice, yes/no) and open-ended formats. GPT-4 was used to help generate and evaluate open-ended question-answer pairs.
Datasets
CARES is built upon data from seven sources:
- Medical Vision-Language Datasets: MIMIC-CXR, IU-Xray, Harvard-FairVLMed, PMC-OA.
- Medical Image Classification Datasets: HAM10000, OL3I.
- Large-Scale VQA Dataset: OmniMedVQA.
These datasets cover a wide range of modalities (X-ray, MRI, CT, Pathology, Fundus, Dermoscopy, etc.) and anatomical regions (chest, lung, eye, skin, etc.). The benchmark itself comprises around 18,000 images and 41,000 QA pairs.
Key Insights and Ideas
- The "Trustworthiness Gap": The most striking insight is that current Med-LVLMs, despite their impressive capabilities, are far from trustworthy for clinical deployment. They consistently fall short across multiple critical dimensions.
- Factual Inaccuracies are Rampant: Models frequently "hallucinate" information, providing incorrect diagnoses or descriptions, especially with rare modalities or open-ended questions. This is a major red flag for patient safety.
- Fairness is a Significant Concern: Performance disparities exist across age, gender, and race, with models often performing worse for the elderly and certain racial groups. This could exacerbate existing health inequities.
- Vulnerability to Attacks: Med-LVLMs show poor resistance to jailbreaking prompts and can be induced to produce toxic or incorrect advice, posing security risks.
- Privacy is Largely Ignored: Models demonstrate a significant lack of awareness regarding patient privacy, with some even leaking information from input prompts.
- Overcautiousness vs. Robustness Trade-off: Some models, like LLaVA-Med, exhibit strong safety measures by refusing to answer, but this can lead to overcautiousness, hindering usability even for routine queries. Conversely, many models are not robust to noisy data or OOD inputs.
Key Results
- Factuality: Most models achieved less than 50% accuracy on the factuality evaluation.
- Fairness: Age-related disparities were notable, with models performing best in the 40-60 age group and declining for the elderly. Racial disparities were also observed, with better performance for Caucasian/Hispanic populations.
- Safety: All models showed reduced accuracy under jailbreaking attacks. LLaVA-Med demonstrated better resistance by abstaining from some attacks, but also exhibited severe overcautiousness.
- Privacy: Models showed significant shortfalls in privacy protection, with low abstention rates on privacy-related questions.
- Robustness: Models exhibited poor OOD robustness, with low abstention rates even when presented with noisy or out-of-distribution data, indicating a tendency to guess rather than refuse.
Limitations and Future Work
The authors acknowledge several limitations:
- Data Coverage: While CARES covers many modalities and regions, limitations in available open-source medical data mean not all areas are fully represented.
- Data Leakage: Despite efforts to use only test sets, there's a possibility that some data might have been included in the pre-training corpora of the evaluated LLMs.
- Evaluation Scope: While CARES covers five key dimensions, other aspects of trustworthiness, such as ethical considerations, could be explored further.
Future work suggested includes:
- Clinical Expert Assessment: Incorporating human expert evaluations for more nuanced and accurate assessments.
- Expanded Evaluation Dimensions: Exploring additional trustworthiness aspects like ethical considerations.
- Richer Data Sources: As more medical data becomes available, expanding the benchmark to cover more modalities and anatomical sites.
- More SOTA Models: Continuously evaluating newer and more advanced Med-LVLMs as they emerge.
Summary updated