Medical Knowledge Intervention Prompt Tuning for Medical Image Classification

Ye Du, Nanxi Yu, Shujun WangIEEE TMI 2025 · DOI

TL;DR

This paper introduces CILMP, a novel prompt tuning method that injects specialized medical knowledge from Large Language Models (LLMs) into Vision-Language Models (VLMs) for medical image classification. By adaptively intervening in LLM representations based on individual images, CILMP significantly improves classification accuracy across diverse medical datasets, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) have shown great promise in transferring knowledge and generalizing across various tasks, including medical image analysis. However, fine-tuning these massive models is computationally expensive. Prompt tuning offers a parameter-efficient alternative by training only a small set of learnable tokens (prompts) while keeping the VLM frozen.

Despite its efficiency, traditional prompt tuning struggles with medical image classification because it often relies on generic class names, failing to capture the nuanced, disease-specific visual features crucial for accurate diagnosis. For instance, distinguishing between "viral pneumonia" and "bacterial pneumonia" based solely on class names is difficult due to their textual similarity. This paper addresses this gap by proposing a method that leverages the rich medical knowledge embedded in LLMs to create more discriminative and instance-adaptive prompts for VLMs.

Methods

The core of the proposed method, CILMP (Conditional Intervention of Large Language Models for Prompt Tuning), involves several key steps:

  • Concept-aware Representation Generation: CILMP queries an LLM (e.g., LLaMA-3-8B) with questions about the distinctive visual features of a specific disease (e.g., "In an image, describe the distinctive visual features of basal cell carcinoma"). This generates concept-aware representations from the LLM.
  • Adaptive Disease Prompts Construction: To bridge the domain gap between LLMs and VLMs, CILMP introduces an adaptation function that maps LLM representations into the VLM feature space. These adapted representations are then concatenated with the original class prompts. A low-rank decomposition is used for parameter efficiency.
  • Conditional Intervention Mechanism: This is the most novel aspect. Instead of a static intervention, CILMP conditions the intervention process on a "Relationship Descriptor" (RD) calculated between the image feature and the LLM representation. This RD captures modality compatibility and allows the prompts to be instance-adaptive, meaning they are tailored to each individual medical image. This conditional mechanism is inspired by Representation Fine-Tuning (ReFT).
  • Model Optimization: The VLM parameters are frozen, and the model is trained using a supervised contrastive loss to optimize the generated adaptive disease prompts.

Datasets

The authors evaluated CILMP on 11 diverse medical image datasets spanning six modalities:

  • Dermatoscope: DermaMNIST, Derm7pt, ISIC 2018
  • Fundus: ADAM, APTOS 2019, ODIR
  • Ultrasound: Fetal-US
  • Histopathology: Chaoyang
  • Endoscope: Kvasir
  • X-ray: CPN-X-ray, Pneumonia

Key Insights and Ideas

  • LLMs as Medical Knowledge Sources: The paper convincingly demonstrates that LLMs, trained on vast text corpora, possess rich, specialized medical knowledge that can be effectively extracted and utilized to improve VLM performance on medical tasks.
  • Instance-Adaptive Prompts are Crucial: The key innovation is conditioning prompt generation on individual images. This allows CILMP to create prompts that are not only disease-specific but also tailored to the unique visual characteristics of each image, leading to better discrimination.
  • Bridging the Domain Gap: CILMP effectively bridges the gap between LLM-generated knowledge and VLM representations through a low-rank linear subspace intervention, maintaining parameter efficiency.
  • Parameter Efficiency: CILMP achieves state-of-the-art performance with a significantly smaller number of trainable parameters compared to many other prompt tuning methods and vastly fewer than full fine-tuning.

Key Results

  • CILMP consistently outperformed 15 state-of-the-art prompt tuning methods on 10 out of 11 diverse medical image datasets.
  • On average across 11 datasets, CILMP achieved superior performance in accuracy, F1-score, AUC, and Kappa score compared to the next best method (DCPL).
  • For example, on the challenging ODIR dataset, CILMP outperformed the CoPrompt method by 0.9% in accuracy and 2.3% in F1-score.
  • Qualitative analysis using t-SNE visualization showed that CILMP generates more discriminative features across classes, particularly for challenging cases like pneumonia subtypes.
  • The method demonstrated robustness in data-efficient scenarios, achieving reasonable performance even with only 10% of labeled data.

Limitations and Future Work

  • Computational Cost of LLM Inference: While CILMP is parameter-efficient for the VLM, it still requires inference from a large LLM to extract knowledge, which can be computationally demanding.
  • Interpretability: The medical knowledge is embedded in latent representations, making it less transparent and potentially hindering clinical acceptance. Future work aims to enhance interpretability using XAI techniques.
  • Auditing Clinical Knowledge: Ensuring that the encoded clinical knowledge accurately aligns with expert understanding, especially for rare diseases, remains a challenge.
  • 2D Focus: The current work is limited to 2D medical image classification. Future research could explore its application in 3D imaging or video-based diagnostics.
  • Multi-modal Fusion: Integrating patient history, lab results, and genomic data alongside imaging could further enhance diagnostic performance.
  • Addressing Outliers: The qualitative analysis showed some misclassifications, suggesting that incorporating image-level feature modeling could further improve performance.

Summary updated

BibTeX
@article{du2025medical,
  author = {Ye Du and Nanxi Yu and Shujun Wang},
  title = {{Medical Knowledge Intervention Prompt Tuning for Medical Image Classification}},
  journal = {IEEE TMI},
  year = {2025},
  doi = {10.1109/tmi.2025.3584841},
}
Figure