TL;DR
EPVT (Environment-aware Prompt Vision Transformer) combats the tendency of skin lesion classifiers to overfit to disease-irrelevant image artifacts like hairs, rulers, or dark corners. By combining domain-specific and shared prompts via a low-rank prompt generator, EPVT effectively disentangles clinical features from environmental noise. This approach achieves state-of-the-art domain generalization across multiple unseen clinical and dermoscopic datasets.
What Problem Does This Paper Address?
Deep learning models have shown immense promise in computer-aided diagnosis (CAD) for skin cancer. However, these models are notorious for exploiting "spurious cues"—disease-irrelevant artifacts frequently present in medical datasets, such as calibration rulers, hair, gel bubbles, and dark camera corners.
When a model overfits to these artifacts (e.g., associating the presence of a ruler with a higher probability of malignancy because doctors tend to measure larger tumors), it fails dramatically when deployed in real-world environments where these artifacts are absent or altered.
Learning domain-invariant features in medical imaging is exceptionally difficult due to the immense diversity in lesion shapes, colors, and textures. Previous domain generalization (DG) methods often relied on weak domain experts (like batch normalization statistics) or failed to facilitate collaborative knowledge sharing between domains, leaving models vulnerable to distribution shifts.
Methods
The authors introduce EPVT, an environment-aware prompt vision transformer that treats different types of image artifacts as distinct "source domains."
The architecture consists of three core components:
- Domain-specific Prompt Learning: Built on top of a Vision Transformer (ViT) backbone, EPVT prepends learnable prompt tokens to the input sequence. Each domain (defined by a specific artifact type) gets its own prompt, forcing it to act as a specialized "domain expert."
- Cross-domain Knowledge Learning (Domain Prompt Generator): To prevent domain prompts from learning completely isolated representations, the authors introduce a low-rank prompt generator. Each domain prompt ($P^m$) is decomposed into a Hadamard (element-wise) product of a globally shared prompt ($P^*$) and a domain-specific rank-one matrix ($P_k = u_k v_k^T$). This structure encourages cross-domain information sharing while preserving unique domain characteristics.
- Domain Mixup Strategy: Real-world images often contain multiple overlapping artifacts (e.g., both hair and a dark corner). EPVT employs a domain mixup strategy during training, blending images from different domains and applying soft cross-entropy losses to smooth out decision margins and handle ambiguous domain labels.
During inference, a lightweight Adapter module dynamically estimates the linear correlation between the target image's features and the source domain prompts, combining them into an optimized, adapted prompt for final prediction.
Datasets
The model was trained and evaluated using several prominent skin lesion benchmarks:
- Source/Training Dataset: ISIC2019, partitioned into five distinct domains based on artifact annotations: dark corner (2,351 images), hair (4,884 images), gel bubble (1,640 images), ruler (672 images), and clean (2,796 images).
- Out-of-Distribution (OOD) Target Datasets:
- Derm7pt-Dermoscopic & Derm7pt-Clinical (evaluating shifts between dermoscopic and standard smartphone clinical images).
- PH2 (dermoscopic images).
- PAD-UFES-20 (clinical smartphone images).
- Trap Sets: Six synthetic splits derived from ISIC2019 with artificially injected bias levels ranging from 0 (no bias) to 1 (maximum adversarial bias, where artifacts strongly correlate with the wrong class label).
Key Insights and Ideas
- Artifacts as Domains: Instead of attempting to pre-process and remove artifacts (which can inadvertently destroy clinical information), EPVT explicitly embraces them as domain labels. This allows the transformer's self-attention mechanism to model the relationship between artifact tokens and image patches, effectively learning what to ignore.
- Low-Rank Multiplicative Updates: Borrowing insights from parameter-efficient fine-tuning in NLP, the authors show that domain-specific updates possess a low intrinsic rank. Decomposing prompts into rank-one subspaces ensures efficient parameter usage and prevents overfitting.
- Intelligent Prompt Weighting: The adapter successfully learns a negative correlation between domain distance and prompt weights. For example, when evaluating on the Derm7pt-Clinical dataset, the model automatically assigns the highest weight to the "dark corner" domain prompt because it shares the closest feature distribution, validating that the adapter behaves logically.
Key Results
EPVT demonstrates superior robustness and generalization performance across the board:
- OOD Generalization: As shown in the results table below, EPVT achieved the highest average ROC-AUC of 84.11% across the four unseen target datasets, outperforming standard Empirical Risk Minimization (ERM) by 4.48% and beating the previous best DG method (DoPrompt) by 2.05%.
- Debiasing Performance: On the most severe adversarial "trap set" (bias degree 1), standard ERM performance collapsed to 65.2% AUC due to its reliance on spurious cues. EPVT maintained a robust 74.6% AUC, outperforming the baseline by 9.4%.
Limitations and Future Work
The primary limitation of EPVT is its reliance on explicit artifact annotations to define source domains during the training phase. Manually labeling thousands of images for dark corners, hairs, and rulers is labor-intensive and may not scale to all medical imaging modalities.
The authors suggest that future research should focus on unsupervised or self-supervised discovery of artifact-based domains to make the pipeline fully automated.
Summary updated