TL;DR
BioVLM is a novel prompt-learning framework designed to help biomedical vision-language models (VLMs) generalize across diverse and unseen medical imaging modalities without fine-tuning their core parameters. By maintaining a diverse bank of LLM-generated clinical prompts and dynamically routing the most informative ones using a low-entropy selection strategy, BioVLM achieves state-of-the-art performance across multiple medical imaging benchmarks while remaining highly parameter-efficient.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) like CLIP have revolutionized zero-shot image classification, and specialized biomedical versions (e.g., BioMedCLIP) have been trained on massive clinical image-text datasets. However, medical imaging presents a unique challenge: a model might perform beautifully on chest X-rays but fail completely on ultrasound or MRI scans due to massive variations in signal formation, textures, and scanner artifacts.
Existing parameter-efficient adaptation methods, such as prompt tuning (e.g., CoOp, BioMedCoOp), typically try to optimize a single static prompt or average multiple text descriptions into one vector. This "one-size-fits-all" approach dilutes fine-grained clinical details and struggles to handle the sheer diversity of biomedical modalities. Test-time adaptation methods like TPT rely on image augmentations (like cropping or zooming), which can inadvertently destroy subtle diagnostic evidence in medical images.
Methods
BioVLM shifts the paradigm from adapting model parameters to dynamically routing text prompts. Its architecture consists of three main components:
- Diverse Prompt Pool Generation: Instead of a single generic prompt, the authors use an LLM (GPT-4o) to generate multiple highly detailed, visual, and clinical attributes for each medical condition. Each attribute is paired with a unique learnable prompt vector, creating a comprehensive "prompt bank" per class.
- Entropy-Based Prompt Selection: At inference time, an image is processed by the frozen vision encoder. Instead of using all prompts, BioVLM evaluates how well each prompt variant aligns with the image. Prompts that align well yield sharp, high-confidence class distributions with low self-entropy. The framework dynamically selects only the prompts falling below a percentile-based entropy threshold ($\tau_\rho$) to make the final prediction.
- Robust Cross-Modal Alignment & Distillation: During training, the model enforces a strict one-to-one mapping between learnable prompts and LLM attributes ($L_{ASA}$). To ensure the prompts capture robust visual features rather than superficial noise, the authors implement a dual strong/weak image augmentation consistency strategy alongside cross-modal knowledge distillation ($L_{CMD}$).
[Input Image] ---> [Frozen Vision Encoder] ---> Visual Embedding (V)
|
(Low-Entropy Selection Router)
|
[Prompt Bank] ---> [Frozen Text Encoder] ----> [Selected High-Confidence Prompts] ---> Final Prediction
Datasets
The authors rigorously evaluated BioVLM across two extensive benchmarks:
- MedMNIST+ Benchmark: Consists of 11 diverse 2D datasets covering 9 imaging modalities (including pathology slides, dermoscopy, retinal OCT, chest X-rays, blood cell microscopes, and abdominal CT scans).
- Biomedical Benchmark: A collection of 11 independent datasets used in prior literature (including CTKidney, Kvasir for endoscopy, RETINA for fundus photography, and BTMRI for brain tumors).
Key Insights and Ideas
- Routing Prompts, Not Parameters: Forcing a VLM's text encoder to compress the vast diversity of medical imaging into a single prompt vector causes a bottleneck. Maintaining a diverse pool and routing the right prompt to the right image preserves fine-grained discriminative power.
- Entropy as a Quality Filter: Self-entropy of the predictive distribution serves as an elegant, unsupervised proxy for prompt alignment. If a prompt yields high entropy (uncertainty), it is safely discarded at test time, filtering out noisy or irrelevant LLM attributes.
- One-to-One Mapping Prevents Feature Washing: Prior methods like BioMedCoOp averaged multiple text descriptions together. BioVLM proves that enforcing a strict one-to-one alignment between specific learnable prompts and specific clinical attributes prevents feature dilution and boosts generalization to unseen categories.
Key Results
BioVLM was tested across three distinct generalization settings and consistently outperformed all prompt-learning and parameter-efficient fine-tuning (PEFT) baselines:
- Few-Shot Learning (FSL): On the MedMNIST+ benchmark, BioVLM achieved an average accuracy of 56.51%, outperforming the previous state-of-the-art biomedical baseline, BioMedCoOp (41.37%), by a massive 15.14% absolute margin. It showed an extraordinary +27.37% gain specifically on the BloodMNIST dataset.
- Base-to-New (B2N) Generalization: When evaluating the model's ability to recognize completely unseen categories within a known modality, BioVLM achieved the highest harmonic mean of 51.77%, outperforming baselines by an average of +8.04%.
- Out-of-Distribution (OOD) Generalization: When trained on a single source dataset and evaluated across all other unseen domains, BioVLM maintained the highest overall robustness (32.34% average accuracy).
- Parameter Efficiency: Despite its superior performance, BioVLM only requires 0.0307 million trainable parameters, which is significantly lighter than multi-modal prompting methods like MaPLe (5.346M) or TCP (0.496M).
Limitations and Future Work
- LLM Dependency: The framework relies heavily on offline LLM attribute generation. If the LLM generates inaccurate or clinically flawed descriptions, it could negatively impact prompt learning.
- Dimensionality Constraints: The current study is strictly focused on 2D medical images. Extending this framework to 3D volumetric data (like full CT or MRI scans) remains an open challenge that will require additional architectural and computational considerations.
- Training Complexity: Managing multiple learnable prompts per class increases optimization complexity during the training phase, which could impact scalability in massive, multi-class clinical environments.
Summary updated