Multimodal Carotid Risk Stratification with Large Vision-Language Models: Benchmarking, Fine-Tuning, and Clinical Insights

Daphne Tsolissou, Theofanis Ganitidis, Konstantinos Mitsis, Stergios CHristodoulidis, Maria Vakalopoulou, Konstantina S. Nikita

Abstract

Accurately assessing stroke risk from carotid plaques requires the integration of complex imaging features with diverse clinical biomarkers. While traditional deep learning approaches rely on labor-intensive manual segmentations and specialized architectures, Large Vision-Language Models (LVLMs) offer a unified framework for multimodal reasoning. This paper benchmarks several state-of-the-art LVLMs on carotid ultrasound risk stratification. We demonstrate that while general-purpose models fail at zero-shot risk classification, a fine-tuned LLaVa-NeXT model—leveraging Low-Rank Adaptation (LoRA) and textualized tabular data—achieves a patient-level AUC of 84.3%. This performance significantly outperforms traditional CNN-based baselines, highlighting the potential for LVLMs to serve as powerful clinical decision-support tools.

Introduction

Stroke risk assessment via carotid ultrasound imaging (USI) is a cornerstone of cardiovascular medicine. However, interpreting these "noisy" images is challenging, often requiring clinicians to synthesize visual plaque characteristics with a patient’s medical history, including age, symptoms, and laboratory biomarkers.

Traditional AI approaches in this domain have largely focused on specialized Convolutional Neural Networks (CNNs). While effective, these models typically require manual segmentation of the carotid artery—a time-consuming process—and struggle to natively integrate non-imaging clinical data. Large Vision-Language Models (LVLMs) present a paradigm shift, as they are designed to process visual and textual information simultaneously. This study investigates whether general-purpose LVLMs can be adapted for specialized cardiovascular risk stratification and evaluates the impact of multimodal clinical context on diagnostic accuracy.

Methodology

The research team evaluated six open-source LVLMs, including general-purpose architectures such as LLaVa-NeXT and PaliGemma, alongside the medically-tuned MedGemma. The evaluation was conducted in two distinct phases:

1. Zero-Shot Benchmarking

Models were subjected to an "interview-style" prompting sequence. Without any prior training on the specific dataset, the models were asked to identify the imaging modality, the anatomy shown, and the patient's risk level. This phase aimed to assess the baseline medical knowledge and visual recognition capabilities of "off-the-shelf" models.

2. Model Adaptation (Fine-Tuning)

The highest-performing general model, LLaVa-NeXT-Vicuna-7B, was selected for fine-tuning using Low-Rank Adaptation (LoRA). Two key strategies were employed:

  • Multimodal Integration: Tabular clinical data (e.g., demographics, LDL levels, and protein biomarkers) were converted into natural language strings and concatenated with the image prompts.
  • Reasoning Chains: The models were trained using Chain-of-Thought (CoT) prompting, guiding the system through a step-by-step diagnostic analysis before reaching a final risk classification.

Experimental Setup

The study utilized an anonymized private dataset of 72 patients, categorized into high-risk (59) and low-risk (13) groups. "High risk" was defined as having >70% stenosis or a recent history of stroke/TIA (within six months). The data included:

  • B-mode Carotid Ultrasound: Static frames sampled from ultrasound videos.
  • Clinical Data: Demographics, laboratory results, and specific protein biomarkers.

Results and Clinical Insights

Zero-Shot Performance and the "Label Trap"

In the zero-shot phase, models like MedGemma (98.3%) and LLaVa-NeXT-7B (91.9%) showed high accuracy in identifying the carotid artery. However, qualitative analysis revealed a critical nuance: models often identified the anatomy by "reading" textual labels (e.g., "ICA" for Internal Carotid Artery) embedded in the images by technicians, rather than through pure visual analysis of the arterial structure.

Furthermore, all models failed the zero-shot risk classification task. They exhibited a significant bias toward predicting "High Risk" regardless of the visual evidence, confirming that general-purpose medical knowledge is insufficient for specialized diagnostic tasks without domain-specific adaptation.

Fine-Tuning and Multimodal Synergy

Fine-tuning yielded substantial performance gains. The LLaVa-NeXT model achieved a patient-level AUC of 82.6% using images alone. When clinical tabular data was integrated as text, the AUC rose to 84.3%, with a Balanced Accuracy of 77.5%.

Crucially, the addition of clinical context significantly improved the model's specificity. In medical AI, the ability to correctly identify low-risk patients is often a bottleneck; these results suggest that multimodal inputs provide the necessary context to reduce false positives. Notably, the fine-tuned LVLM outperformed a prior CNN-Ensemble baseline (AUC of 73%) that had been trained on the same dataset using expert-annotated segments.

Scaling vs. Architecture

An interesting finding was that the 7B-parameter version of LLaVa-NeXT outperformed the larger 13B-parameter version in modality detection. This suggests that for specific medical imaging tasks, the underlying architecture and the quality of the tuning process may be more influential than raw parameter count.

Discussion and Future Work

While the results are promising, several limitations remain. The dataset size (72 patients) and class imbalance introduce variance in specificity across different validation folds. Additionally, the current model processes static frames, whereas clinicians often rely on temporal cues—such as plaque stability during the cardiac cycle—visible in ultrasound videos.

Future research will focus on:

  1. Video-based LVLMs: Incorporating temporal information to better mimic clinical workflows.
  2. Model Calibration: Improving the reliability of probability scores to ensure clinical safety.
  3. Expanded Datasets: Validating the approach on larger, more diverse patient cohorts to improve generalization.

Conclusion

This study demonstrates that fine-tuned LVLMs can effectively bridge the gap between complex medical imaging and clinical tabular data. By outperforming traditional CNNs without requiring manual segmentations, these models offer a scalable and highly accurate path forward for automated carotid risk stratification.

Summary updated

Abstract

Reliable risk assessment for carotid atheromatous disease remains a major clinical challenge, as it requires integrating diverse clinical and imaging information in a manner that is transparent and interpretable to clinicians. This study investigates the potential of state-of-the-art and recent large vision-language models (LVLMs) for multimodal carotid plaque assessment by integrating ultrasound imaging (USI) with structured clinical, demographic, laboratory, and protein biomarker data. A framework that simulates realistic diagnostic scenarios through interview-style question sequences is proposed, comparing a range of open-source LVLMs, including both general-purpose and medically tuned models. Zero-shot experiments reveal that even if they are very powerful, not all LVLMs can accurately identify imaging modality and anatomy, while all of them perform poorly in accurate risk classification. To address this limitation, LLaVa-NeXT-Vicuna is adapted to the ultrasound domain using low-rank adaptation (LoRA), resulting in substantial improvements in stroke risk stratification. The integration of multimodal tabular data in the form of text further enhances specificity and balanced accuracy, yielding competitive performance compared to prior convolutional neural network (CNN) baselines trained on the same dataset. Our findings highlight both the promise and limitations of LVLMs in ultrasound-based cardiovascular risk prediction, underscoring the importance of multimodal integration, model calibration, and domain adaptation for clinical translation.
BibTeX
@inproceedings{tsolissou2025multimodal,
  author = {Daphne Tsolissou and Theofanis Ganitidis and Konstantinos Mitsis and Stergios CHristodoulidis and Maria Vakalopoulou and Konstantina S. Nikita},
  title = {{Multimodal Carotid Risk Stratification with Large Vision-Language Models: Benchmarking, Fine-Tuning, and Clinical Insights}},
  booktitle = {BIBE},
  year = {2025},
  doi = {10.1109/bibe66822.2025.00122},
}
Figure