Abstract
Accurately assessing stroke risk from carotid plaques requires the integration of complex imaging features with diverse clinical biomarkers. While traditional deep learning approaches rely on labor-intensive manual segmentations and specialized architectures, Large Vision-Language Models (LVLMs) offer a unified framework for multimodal reasoning. This paper benchmarks several state-of-the-art LVLMs on carotid ultrasound risk stratification. We demonstrate that while general-purpose models fail at zero-shot risk classification, a fine-tuned LLaVa-NeXT model—leveraging Low-Rank Adaptation (LoRA) and textualized tabular data—achieves a patient-level AUC of 84.3%. This performance significantly outperforms traditional CNN-based baselines, highlighting the potential for LVLMs to serve as powerful clinical decision-support tools.
Introduction
Stroke risk assessment via carotid ultrasound imaging (USI) is a cornerstone of cardiovascular medicine. However, interpreting these "noisy" images is challenging, often requiring clinicians to synthesize visual plaque characteristics with a patient’s medical history, including age, symptoms, and laboratory biomarkers.
Traditional AI approaches in this domain have largely focused on specialized Convolutional Neural Networks (CNNs). While effective, these models typically require manual segmentation of the carotid artery—a time-consuming process—and struggle to natively integrate non-imaging clinical data. Large Vision-Language Models (LVLMs) present a paradigm shift, as they are designed to process visual and textual information simultaneously. This study investigates whether general-purpose LVLMs can be adapted for specialized cardiovascular risk stratification and evaluates the impact of multimodal clinical context on diagnostic accuracy.
Methodology
The research team evaluated six open-source LVLMs, including general-purpose architectures such as LLaVa-NeXT and PaliGemma, alongside the medically-tuned MedGemma. The evaluation was conducted in two distinct phases:
1. Zero-Shot Benchmarking
Models were subjected to an "interview-style" prompting sequence. Without any prior training on the specific dataset, the models were asked to identify the imaging modality, the anatomy shown, and the patient's risk level. This phase aimed to assess the baseline medical knowledge and visual recognition capabilities of "off-the-shelf" models.
2. Model Adaptation (Fine-Tuning)
The highest-performing general model, LLaVa-NeXT-Vicuna-7B, was selected for fine-tuning using Low-Rank Adaptation (LoRA). Two key strategies were employed:
- Multimodal Integration: Tabular clinical data (e.g., demographics, LDL levels, and protein biomarkers) were converted into natural language strings and concatenated with the image prompts.
- Reasoning Chains: The models were trained using Chain-of-Thought (CoT) prompting, guiding the system through a step-by-step diagnostic analysis before reaching a final risk classification.
Experimental Setup
The study utilized an anonymized private dataset of 72 patients, categorized into high-risk (59) and low-risk (13) groups. "High risk" was defined as having >70% stenosis or a recent history of stroke/TIA (within six months). The data included:
- B-mode Carotid Ultrasound: Static frames sampled from ultrasound videos.
- Clinical Data: Demographics, laboratory results, and specific protein biomarkers.
Results and Clinical Insights
Zero-Shot Performance and the "Label Trap"
In the zero-shot phase, models like MedGemma (98.3%) and LLaVa-NeXT-7B (91.9%) showed high accuracy in identifying the carotid artery. However, qualitative analysis revealed a critical nuance: models often identified the anatomy by "reading" textual labels (e.g., "ICA" for Internal Carotid Artery) embedded in the images by technicians, rather than through pure visual analysis of the arterial structure.
Furthermore, all models failed the zero-shot risk classification task. They exhibited a significant bias toward predicting "High Risk" regardless of the visual evidence, confirming that general-purpose medical knowledge is insufficient for specialized diagnostic tasks without domain-specific adaptation.
Fine-Tuning and Multimodal Synergy
Fine-tuning yielded substantial performance gains. The LLaVa-NeXT model achieved a patient-level AUC of 82.6% using images alone. When clinical tabular data was integrated as text, the AUC rose to 84.3%, with a Balanced Accuracy of 77.5%.
Crucially, the addition of clinical context significantly improved the model's specificity. In medical AI, the ability to correctly identify low-risk patients is often a bottleneck; these results suggest that multimodal inputs provide the necessary context to reduce false positives. Notably, the fine-tuned LVLM outperformed a prior CNN-Ensemble baseline (AUC of 73%) that had been trained on the same dataset using expert-annotated segments.
Scaling vs. Architecture
An interesting finding was that the 7B-parameter version of LLaVa-NeXT outperformed the larger 13B-parameter version in modality detection. This suggests that for specific medical imaging tasks, the underlying architecture and the quality of the tuning process may be more influential than raw parameter count.
Discussion and Future Work
While the results are promising, several limitations remain. The dataset size (72 patients) and class imbalance introduce variance in specificity across different validation folds. Additionally, the current model processes static frames, whereas clinicians often rely on temporal cues—such as plaque stability during the cardiac cycle—visible in ultrasound videos.
Future research will focus on:
- Video-based LVLMs: Incorporating temporal information to better mimic clinical workflows.
- Model Calibration: Improving the reliability of probability scores to ensure clinical safety.
- Expanded Datasets: Validating the approach on larger, more diverse patient cohorts to improve generalization.
Conclusion
This study demonstrates that fine-tuned LVLMs can effectively bridge the gap between complex medical imaging and clinical tabular data. By outperforming traditional CNNs without requiring manual segmentations, these models offer a scalable and highly accurate path forward for automated carotid risk stratification.
Summary updated