Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification

Xiaoliu Luo, Minxue Xiao, Ting Xie, Mengzhu Wang, Huiqing Qi, Joey Tianyi Zhou, Taiping Zhang, Xu WangarXiv 2026 · arXiv

TL;DR

The paper introduces Multi-View Synergistic Learning (MVSL), a unified framework designed to adapt biomedical vision-language models for low-resource image classification. By decoupling visual and textual tuning strategies, aligning features at both global and local granularities, and incorporating an LLM-guided disease semantic graph, MVSL significantly improves classification accuracy and interpretability in few-shot and zero-shot medical scenarios.

What Problem Does This Paper Address?

Biomedical image classification often suffers from a severe lack of annotated data due to the high cost of expert medical annotation and the rarity of certain diseases. While large-scale vision-language models (VLMs) like BiomedCLIP offer a strong foundation by learning from massive image-text datasets, effectively adapting them to specific low-resource downstream tasks remains a major hurdle.

Medical images contain subtle, highly localized visual differences (such as distinguishing melanoma from a benign melanocytic nevus), and disease categories share complex semantic relationships. Existing parameter-efficient fine-tuning (PEFT) methods typically apply the same tuning strategy uniformly to both vision and text branches. This uniform treatment ignores the fact that visual data is continuous and spatial, while text is discrete and abstract, leading to unstable cross-modal alignment and poor generalization when training data is scarce.

Methods

MVSL tackles these challenges by integrating three complementary components into a cohesive framework:

  1. Cross-Paradigm Fine-Tuning (CPFT): Instead of a one-size-fits-all approach, CPFT decouples the modalities. It inserts lightweight Vision Residual Adapters (VRA) into the visual encoder to refine spatial and lesion-level features, while using Text Prompt Tuning (learnable soft prompts) on the text encoder to inject task-specific semantic context.
  2. Multi-Granularity Contrastive Learning (MGCL): Standard contrastive learning frameworks focus primarily on global image-text alignment. MGCL explicitly models both global image semantics and localized patch-level features to align with text embeddings, ensuring the model learns to focus on specific lesion areas. A dynamic fusion mechanism adaptively combines these global and local predictions during inference.
  3. Disease Semantic Graph (DSG): To preserve the broader clinical relationships between diseases, the authors leverage a Large Language Model (GPT-4) to generate detailed clinical descriptions for each class. These descriptions are used to construct a semantic graph mapping out the global topology of the disease categories. Through Laplacian structural distillation, this graph acts as a teacher prior to regularize the text embeddings, which indirectly stabilizes the visual embeddings via cross-modal alignment.
  +-----------------------------------------------------------------+
  |                       MVSL Framework                            |
  +-----------------------------------------------------------------+
          |                                 |
          v                                 v
+-----------------------+       +-----------------------------------+
| Visual Branch (ViT)   |       | Textual Branch                    |
| - Upper-layer VRAs    |       | - Learnable Soft Prompts          |
+-----------------------+       +-----------------------------------+
          |                                 |
          +-------->  Global & Local  <-----+
                  Contrastive Alignment (MGCL)
                            ^
                            | (Structural Distillation)
                +-----------------------+
                | Disease Semantic Graph| (LLM-Guided Topology)
                +-----------------------+

Datasets

The framework was comprehensively evaluated on 11 public biomedical datasets spanning 9 imaging modalities and 10 anatomical regions:

  • CTKidney: Kidney CT scans (cysts, stones, tumors).
  • DermaMNIST: Dermatoscopy skin lesions.
  • Kvasir: Endoscopy colon images.
  • RETINA: Fundus photography for eye diseases.
  • LC25000 & CHMNIST: Lung/colon and colorectal histopathology.
  • BTMRI: Brain MRI tumor scans.
  • OCTMNIST: Optical Coherence Tomography retina scans.
  • BUSI: Breast Ultrasound scans.
  • COVID-QU-Ex: Chest X-rays.
  • KneeXray: Knee osteoarthritis severity grading.

No new benchmarks were introduced; the authors adopted the evaluation suite established by prior state-of-the-art methods like BiomedCoOp to ensure a fair comparison.

Key Insights and Ideas

  • Asymmetry is a Feature, Not a Bug: Treating vision and language encoders differently during fine-tuning yields better results. Visual encoders benefit more from structural architectural modifications (adapters), whereas text encoders respond better to contextual prompt adjustments.
  • Upper Layers Matter Most for Adapters: Ablation studies revealed that inserting adapters only into the final layers of the vision transformer (specifically Block 11) provides the optimal balance. Stacking adapters across all layers increases computational burden and actually degrades generalization due to representation drift.
  • Local Alignment Needs Global Structure: While patch-level contrastive learning helps find small lesions, it can cause overfitting to base classes. Introducing the Disease Semantic Graph counteracts this by enforcing global class-level relationships, proving that local visual cues and global semantic structures are mutually essential.

Key Results

  • State-of-the-Art Few-Shot Performance: MVSL consistently outperformed 16 baseline methods across all few-shot settings ($K = 1, 2, 4, 8, 16$). Averaged across all 11 datasets, MVSL achieved 58.41% accuracy at 1-shot (a +1.38% improvement over the previous best, BiomedCoOp) and 77.13% at 16-shot (a +3.72% improvement).
  • Robust Base-to-Novel Generalization: In zero-shot generalization tests (training on base classes and testing on unseen novel classes), MVSL achieved a Harmonic Mean (HM) score of 77.22%, outperforming BiomedCoOp (76.29%) and significantly beating standard prompt-tuning methods like CoOp (67.23%).
  • Enhanced Interpretability: Qualitative analysis using gScoreCAM showed that MVSL generates much sharper, more clinically accurate visual saliency maps, focusing precisely on actual pathological regions (e.g., tumors or lung lesions) rather than spreading across irrelevant tissue.

Limitations and Future Work

Not explicitly discussed by the authors in the paper. However, apparent directions for future work include exploring the integration of larger, more advanced vision-language architectures, and testing the framework on broader clinical tasks beyond classification, such as multi-label diagnosis or zero-shot lesion segmentation.

Summary updated

Abstract

Accurate biomedical image classification under low-resource conditions remains challenging due to limited annotations, subtle inter-class visual differences, and complex disease semantics. While vision--language models offer a promising foundation for mitigating data scarcity, their effective adaptation in biomedical settings is constrained by the need for parameter-efficient tuning alongside fine-grained and semantically consistent representation learning. In this work, we propose Multi-View Synergistic Learning (MVSL), a unified framework that addresses these challenges by jointly considering adaptation paradigms, representation granularity, and disease semantic relationships. MVSL decouples the adaptation of visual and textual encoders to respect their distinct representational characteristics, enabling more stable and effective parameter-efficient fine-tuning. It further introduces multi-granularity contrastive learning to explicitly model both global image semantics and localized lesion-level evidence, improving fine-grained discrimination for visually similar disease categories. In addition, MVSL preserves disease-level semantic structure by incorporating structured supervision derived from large language models, which constrains textual representations at the class level and indirectly regularizes visual embeddings through cross-modal alignment. Together, these components enable more stable cross-modal alignment and improved discrimination under limited supervision. Extensive experiments on $11$ public biomedical datasets spanning $9$ imaging modalities and $10$ anatomical regions demonstrate that MVSL consistently outperforms state-of-the-art methods in few-shot and zero-shot classification settings.
BibTeX
@misc{luo2026multi,
  author = {Xiaoliu Luo and Minxue Xiao and Ting Xie and Mengzhu Wang and Huiqing Qi and Joey Tianyi Zhou and Taiping Zhang and Xu Wang},
  title = {{Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification}},
  year = {2026},
  eprint = {2604.23977},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.23977},
}

Loading related content...

Figure