Prompt-driven Latent Domain Generalization for Medical Image Classification

Siyuan Yan, Chi Liu, Zhen Yu, Lie Ju, Dwarikanath Mahapatra, Brigid Betz-Stablein, Victoria Mar, Monika Janda, Peter Soyer, Zongyuan Ge

TL;DR

Medical AI often fails when encountering new hospitals or camera types because it overfits to "easy" artifacts like rulers or lighting styles. This paper introduces Prompt-driven Latent Domain Generalization (PLDG), a framework that automatically discovers these hidden "domains" without manual labels and uses specialized Vision Transformer prompts to ensure the model learns robust, generalizable features.

What Problem Does This Paper Address?

Deep learning models are notoriously brittle when faced with distribution shifts. In medical imaging, a model trained on dermoscopic images from one clinic might fail at another because of subtle differences: a different brand of camera, varying skin tones, or the presence of artifacts like surgical markers, hair, or gel bubbles.

Standard Domain Generalization (DG) techniques try to solve this by training on multiple "domains" (e.g., different hospitals) to learn features that work across all of them. However, DG usually requires domain labels—explicit tags telling the model which image belongs to which hospital or contains which artifact. These labels are expensive to produce, often noisy, and sometimes the "true" domain (like a specific camera sensor's noise pattern) isn't even known to the researchers. The authors tackle the challenge of achieving generalization without needing these pre-defined labels.

Methods

The PLDG framework consists of three main stages, as illustrated in the pipeline below:

The PLDG pipeline: (a) Unsupervised domain discovery via clustering and (b) Prompt-based learning for generalization.
  1. Simplicity Bias-guided Pseudo Domain Discovery: The authors leverage the "Simplicity Bias" property—the tendency of neural networks to learn simple, low-level features (like background colors or textures) before complex semantic ones. By clustering the "CLS" tokens from the shallow layers of a Vision Transformer (ViT) early in training, they can automatically group images into "pseudo-domains" based on style and artifacts rather than medical diagnosis.
  2. Domain-specific Prompt Learning: Once pseudo-domains are identified, the model uses Visual Prompt Tuning. Instead of retraining the whole network for every domain, they insert small, learnable "prompts" (vectors) into the ViT. Each prompt acts as a "domain expert" for one discovered style.
  3. Collaborative Domain Prompt Generator: To prevent these experts from working in silos, the authors use a novel generator that decomposes each domain prompt into a shared component and a domain-specific low-rank matrix. This encourages the prompts to share knowledge while maintaining their unique domain-specific focus.
  4. Domain Mixup & Adapter: During training, they "mix" images from different domains to smooth decision boundaries. At inference time, a small "adapter" network looks at a new, unseen image and decides how to weight the different domain prompts to get the most accurate diagnosis.
Fig. 3. Illustration of (a) domain prompt generator and (b) domain Mixup strategy.

Datasets

The authors benchmarked PLDG across three distinct medical tasks:

  • Melanoma Classification: Trained on ISIC2019 and tested on four out-of-distribution (OOD) sets: Derm7pt (Dermoscopic and Clinical), PH2, and PAD-UFES-20.
  • Diabetic Retinopathy (DR): A combination of EyePACS, APTOS, Messidor, and Messidor 2, where one dataset is held out for testing in a "leave-one-dataset-out" fashion.
  • Cancerous Tissue Detection: The CAMELYON17-WILDS benchmark, which involves histopathology patches from five different hospitals.
  • Debiasing Task: A synthetic "trap" skin dataset designed with increasing levels of correlation between artifacts and labels to test model robustness.

Key Insights and Ideas

  • Shallow Layers are Style Layers: The authors found that the first few layers of a ViT capture global appearance and style, while deeper layers capture category-specific features. Clustering in these shallow layers is the key to finding domains without accidentally clustering by disease type.
  • Latent Domains > Manual Labels: Surprisingly, PLDG often outperformed methods that did use manual domain labels. This suggests that human-defined domains (like "Hospital A vs Hospital B") might be less informative for a model than the latent styles discovered automatically (like "Images with dark corners vs Images with blue tint").
  • Negative Correlation of Distance and Weight: The model successfully learns to assign higher weights to prompts from training domains that are visually "closer" to the unseen test image, proving the adapter effectively recognizes domain similarity.

Key Results

PLDG consistently achieved state-of-the-art or comparable results to supervised DG methods:

  • Melanoma: On the PAD-UFES-20 dataset, PLDG improved ROC-AUC by 14.18% over the standard baseline (ERM) and even beat the version of their own model that used manual labels.
  • Diabetic Retinopathy: PLDG achieved the best average accuracy (67.58%) across all tested OOD datasets, outperforming specialized DG algorithms like DANN and CORAL.
  • Debiasing: In high-bias scenarios (where artifacts strongly mislead the model), PLDG outperformed the baseline by 6.13%, showing significantly better robustness to spurious correlations.

Limitations and Future Work

While powerful, the method relies on the Simplicity Bias assumption; if a dataset's bias is extremely complex or subtle, the initial clustering might fail. The current implementation is also specific to Vision Transformer architectures. The authors suggest that future work could explore more sophisticated clustering methods or adapt the prompt-driven approach to other architectures like CNNs to broaden its applicability in clinical settings where domain labels are perpetually scarce.

Summary updated

Abstract

Deep learning models for medical image analysis easily suffer from distribution shifts caused by dataset artifacts bias, camera variations, differences in the imaging station, etc., leading to unreliable diagnoses in real-world clinical settings. Domain generalization (DG) methods, which aim to train models on multiple domains to perform well on unseen domains, offer a promising direction to solve the problem. However, existing DG methods assume domain labels of each image are available and accurate, which is typically feasible for only a limited number of medical datasets. To address these challenges, we propose a novel DG framework for medical image classification without relying on domain labels, called Prompt-driven Latent Domain Generalization (PLDG). PLDG consists of unsupervised domain discovery and prompt learning. This framework first discovers pseudo domain labels by clustering the bias-associated style features, then leverages collaborative domain prompts to guide a Vision Transformer to learn knowledge from discovered diverse domains. To facilitate cross-domain knowledge learning between different prompts, we introduce a domain prompt generator that enables knowledge sharing between domain prompts and a shared prompt. A domain mixup strategy is additionally employed for more flexible decision margins and mitigates the risk of incorrect domain assignments. Extensive experiments on three medical image classification tasks and one debiasing task demonstrate that our method can achieve comparable or even superior performance than conventional DG algorithms without relying on domain labels. Our code will be publicly available upon the paper is accepted.
BibTeX
@article{yan2025prompt,
  author = {Siyuan Yan and Chi Liu and Zhen Yu and Lie Ju and Dwarikanath Mahapatra and Brigid Betz-Stablein and Victoria Mar and Monika Janda and Peter Soyer and Zongyuan Ge},
  title = {{Prompt-driven Latent Domain Generalization for Medical Image Classification}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2025},
  doi = {10.1109/tmi.2024.3443119},
}
Figure