Introduction
Deep learning models have demonstrated significant potential in the automated diagnosis of skin cancer. However, these models frequently suffer from a "shortcut learning" problem, where they rely on spurious correlations rather than the underlying pathology of the lesion. In clinical datasets, these shortcuts often manifest as disease-irrelevant artifacts, such as surgical markers, ruler markings, gel bubbles, or dense hair.
When a model trained on data containing specific artifacts is deployed in a new environment where those artifacts are absent or different, its performance typically degrades significantly. This challenge is known as a failure of Domain Generalization (DG). While traditional DG methods attempt to extract domain-invariant features, the high variability in skin tones, lighting, and textures makes finding a universal feature set difficult. The Environment-aware Prompt Vision Transformer (EPVT) addresses this by explicitly modeling these artifacts as environmental contexts rather than noise to be ignored.
Methodology
EPVT leverages the Vision Transformer (ViT) architecture and introduces a specialized prompt-tuning framework to manage domain-specific knowledge.
1. Domain-specific Prompts
Rather than training a monolithic model, EPVT treats different artifacts as distinct domains (e.g., "Ruler Domain," "Hair Domain," "Clean Domain"). Each domain is assigned a set of learnable tokens, or prompts, which are prepended to the input image tokens. This allows the model to develop "expert" knowledge for specific environmental conditions.
2. Domain Prompt Generator
To ensure that domain experts can share relevant information, the authors utilize a low-rank multiplicative update strategy. A shared prompt ($P^*$) captures general features across all skin lesions, while a rank-one matrix ($P_k$) captures nuances specific to domain $k$. The final prompt is generated via the Hadamard product: $P^m = P^* \odot P_k$. This approach is parameter-efficient and prevents the experts from working in total isolation.
3. Domain Mixup and Inference Adapter
During training, a Domain Mixup strategy is employed to help the model navigate images containing multiple artifacts (e.g., both hair and rulers). By blending images and labels from different domains, the model learns to handle ambiguous environments. At inference time, an Adapter module calculates a weighted combination of domain prompts based on the input image's features. This allows the model to dynamically select the most relevant "expert" prompts for unseen, out-of-distribution (OOD) images.
Experimental Setup
Datasets
The model was trained on the ISIC2019 dataset, which was manually partitioned into five artifact-based domains: dark corner (vignetting), hair, gel bubble, ruler, and clean.
To evaluate generalization, the authors tested EPVT on several OOD datasets:
- Derm7pt (Dermoscopic and Clinical versions)
- PH2
- PAD-UFES-20 (Smartphone-captured clinical images)
Additionally, "Trap Sets" were created by intentionally correlating artifacts with specific labels in the training data to measure how easily the model could be misled by biased data.
Results and Discussion
Superior Generalization
EPVT achieved an average ROC-AUC of 84.11% across the four OOD datasets, significantly outperforming the standard Empirical Risk Minimization (ERM) baseline (79.63%) and other state-of-the-art DG methods like DoPrompt and SagNet.
Cross-Modality Performance
A key finding was the model's performance on clinical smartphone images (PAD-UFES-20). Despite being trained primarily on dermoscopic images, EPVT showed a 4.1% improvement over the baseline, suggesting that the environment-aware prompts effectively bridge the gap between different imaging modalities.
Resilience to Bias
In the Trap Set experiments, EPVT demonstrated remarkable robustness. At the highest levels of intentional bias—where artifacts were strongly correlated with cancer labels—EPVT outperformed the baseline by 9.4%. This indicates that the model is learning to separate the artifact's influence from the diagnostic features of the lesion, effectively resisting the "shortcut" trap.
Conclusion
EPVT represents a shift in how medical AI handles data artifacts. By treating spurious cues as environmental domains rather than noise, the framework allows for more robust and generalizable skin lesion recognition. While the current implementation relies on manual artifact annotation for training, future iterations may utilize unsupervised domain discovery to further automate the pipeline. Despite the added complexity of managing multiple prompts, the gains in clinical reliability and resilience to bias make EPVT a promising step toward deploying AI in diverse, real-world medical settings.
Summary updated