TL;DR
Medical AI often fails when encountering new hospitals or camera types because it overfits to "easy" artifacts like rulers or lighting styles. This paper introduces Prompt-driven Latent Domain Generalization (PLDG), a framework that automatically discovers these hidden "domains" without manual labels and uses specialized Vision Transformer prompts to ensure the model learns robust, generalizable features.
What Problem Does This Paper Address?
Deep learning models are notoriously brittle when faced with distribution shifts. In medical imaging, a model trained on dermoscopic images from one clinic might fail at another because of subtle differences: a different brand of camera, varying skin tones, or the presence of artifacts like surgical markers, hair, or gel bubbles.
Standard Domain Generalization (DG) techniques try to solve this by training on multiple "domains" (e.g., different hospitals) to learn features that work across all of them. However, DG usually requires domain labels—explicit tags telling the model which image belongs to which hospital or contains which artifact. These labels are expensive to produce, often noisy, and sometimes the "true" domain (like a specific camera sensor's noise pattern) isn't even known to the researchers. The authors tackle the challenge of achieving generalization without needing these pre-defined labels.
Methods
The PLDG framework consists of three main stages, as illustrated in the pipeline below:
- Simplicity Bias-guided Pseudo Domain Discovery: The authors leverage the "Simplicity Bias" property—the tendency of neural networks to learn simple, low-level features (like background colors or textures) before complex semantic ones. By clustering the "CLS" tokens from the shallow layers of a Vision Transformer (ViT) early in training, they can automatically group images into "pseudo-domains" based on style and artifacts rather than medical diagnosis.
- Domain-specific Prompt Learning: Once pseudo-domains are identified, the model uses Visual Prompt Tuning. Instead of retraining the whole network for every domain, they insert small, learnable "prompts" (vectors) into the ViT. Each prompt acts as a "domain expert" for one discovered style.
- Collaborative Domain Prompt Generator: To prevent these experts from working in silos, the authors use a novel generator that decomposes each domain prompt into a shared component and a domain-specific low-rank matrix. This encourages the prompts to share knowledge while maintaining their unique domain-specific focus.
- Domain Mixup & Adapter: During training, they "mix" images from different domains to smooth decision boundaries. At inference time, a small "adapter" network looks at a new, unseen image and decides how to weight the different domain prompts to get the most accurate diagnosis.
Datasets
The authors benchmarked PLDG across three distinct medical tasks:
- Melanoma Classification: Trained on ISIC2019 and tested on four out-of-distribution (OOD) sets: Derm7pt (Dermoscopic and Clinical), PH2, and PAD-UFES-20.
- Diabetic Retinopathy (DR): A combination of EyePACS, APTOS, Messidor, and Messidor 2, where one dataset is held out for testing in a "leave-one-dataset-out" fashion.
- Cancerous Tissue Detection: The CAMELYON17-WILDS benchmark, which involves histopathology patches from five different hospitals.
- Debiasing Task: A synthetic "trap" skin dataset designed with increasing levels of correlation between artifacts and labels to test model robustness.
Key Insights and Ideas
- Shallow Layers are Style Layers: The authors found that the first few layers of a ViT capture global appearance and style, while deeper layers capture category-specific features. Clustering in these shallow layers is the key to finding domains without accidentally clustering by disease type.
- Latent Domains > Manual Labels: Surprisingly, PLDG often outperformed methods that did use manual domain labels. This suggests that human-defined domains (like "Hospital A vs Hospital B") might be less informative for a model than the latent styles discovered automatically (like "Images with dark corners vs Images with blue tint").
- Negative Correlation of Distance and Weight: The model successfully learns to assign higher weights to prompts from training domains that are visually "closer" to the unseen test image, proving the adapter effectively recognizes domain similarity.
Key Results
PLDG consistently achieved state-of-the-art or comparable results to supervised DG methods:
- Melanoma: On the PAD-UFES-20 dataset, PLDG improved ROC-AUC by 14.18% over the standard baseline (ERM) and even beat the version of their own model that used manual labels.
- Diabetic Retinopathy: PLDG achieved the best average accuracy (67.58%) across all tested OOD datasets, outperforming specialized DG algorithms like DANN and CORAL.
- Debiasing: In high-bias scenarios (where artifacts strongly mislead the model), PLDG outperformed the baseline by 6.13%, showing significantly better robustness to spurious correlations.
Limitations and Future Work
While powerful, the method relies on the Simplicity Bias assumption; if a dataset's bias is extremely complex or subtle, the initial clustering might fail. The current implementation is also specific to Vision Transformer architectures. The authors suggest that future work could explore more sophisticated clustering methods or adapt the prompt-driven approach to other architectures like CNNs to broaden its applicability in clinical settings where domain labels are perpetually scarce.
Summary updated