EPVT: Environment-Aware Prompt Vision Transformer for Domain Generalization in Skin Lesion Recognition

Siyuan Yan, Chi Liu, Zhen Yu, Lie Ju, Dwarikanath Mahapatra, Victoria Mar, Monika Janda, H. Peter Soyer, Zongyuan Ge

Introduction

Deep learning models have demonstrated significant potential in the automated diagnosis of skin cancer. However, these models frequently suffer from a "shortcut learning" problem, where they rely on spurious correlations rather than the underlying pathology of the lesion. In clinical datasets, these shortcuts often manifest as disease-irrelevant artifacts, such as surgical markers, ruler markings, gel bubbles, or dense hair.

When a model trained on data containing specific artifacts is deployed in a new environment where those artifacts are absent or different, its performance typically degrades significantly. This challenge is known as a failure of Domain Generalization (DG). While traditional DG methods attempt to extract domain-invariant features, the high variability in skin tones, lighting, and textures makes finding a universal feature set difficult. The Environment-aware Prompt Vision Transformer (EPVT) addresses this by explicitly modeling these artifacts as environmental contexts rather than noise to be ignored.

Methodology

EPVT leverages the Vision Transformer (ViT) architecture and introduces a specialized prompt-tuning framework to manage domain-specific knowledge.

1. Domain-specific Prompts

Rather than training a monolithic model, EPVT treats different artifacts as distinct domains (e.g., "Ruler Domain," "Hair Domain," "Clean Domain"). Each domain is assigned a set of learnable tokens, or prompts, which are prepended to the input image tokens. This allows the model to develop "expert" knowledge for specific environmental conditions.

2. Domain Prompt Generator

To ensure that domain experts can share relevant information, the authors utilize a low-rank multiplicative update strategy. A shared prompt ($P^*$) captures general features across all skin lesions, while a rank-one matrix ($P_k$) captures nuances specific to domain $k$. The final prompt is generated via the Hadamard product: $P^m = P^* \odot P_k$. This approach is parameter-efficient and prevents the experts from working in total isolation.

3. Domain Mixup and Inference Adapter

During training, a Domain Mixup strategy is employed to help the model navigate images containing multiple artifacts (e.g., both hair and rulers). By blending images and labels from different domains, the model learns to handle ambiguous environments. At inference time, an Adapter module calculates a weighted combination of domain prompts based on the input image's features. This allows the model to dynamically select the most relevant "expert" prompts for unseen, out-of-distribution (OOD) images.

Experimental Setup

Datasets

The model was trained on the ISIC2019 dataset, which was manually partitioned into five artifact-based domains: dark corner (vignetting), hair, gel bubble, ruler, and clean.

To evaluate generalization, the authors tested EPVT on several OOD datasets:

  • Derm7pt (Dermoscopic and Clinical versions)
  • PH2
  • PAD-UFES-20 (Smartphone-captured clinical images)

Additionally, "Trap Sets" were created by intentionally correlating artifacts with specific labels in the training data to measure how easily the model could be misled by biased data.

Results and Discussion

Superior Generalization

EPVT achieved an average ROC-AUC of 84.11% across the four OOD datasets, significantly outperforming the standard Empirical Risk Minimization (ERM) baseline (79.63%) and other state-of-the-art DG methods like DoPrompt and SagNet.

Cross-Modality Performance

A key finding was the model's performance on clinical smartphone images (PAD-UFES-20). Despite being trained primarily on dermoscopic images, EPVT showed a 4.1% improvement over the baseline, suggesting that the environment-aware prompts effectively bridge the gap between different imaging modalities.

Resilience to Bias

In the Trap Set experiments, EPVT demonstrated remarkable robustness. At the highest levels of intentional bias—where artifacts were strongly correlated with cancer labels—EPVT outperformed the baseline by 9.4%. This indicates that the model is learning to separate the artifact's influence from the diagnostic features of the lesion, effectively resisting the "shortcut" trap.

Conclusion

EPVT represents a shift in how medical AI handles data artifacts. By treating spurious cues as environmental domains rather than noise, the framework allows for more robust and generalizable skin lesion recognition. While the current implementation relies on manual artifact annotation for training, future iterations may utilize unsupervised domain discovery to further automate the pipeline. Despite the added complexity of managing multiple prompts, the gains in clinical reliability and resilience to bias make EPVT a promising step toward deploying AI in diverse, real-world medical settings.

Summary updated

Abstract

Skin lesion recognition using deep learning has made remarkable progress, and there is an increasing need for deploying these systems in real-world scenarios. However, recent research has revealed that deep neural networks for skin lesion recognition may overly depend on disease-irrelevant image artifacts (i.e., dark corners, dense hairs), leading to poor generalization in unseen environments. To address this issue, we propose a novel domain generalization method called EPVT, which involves embedding prompts into the vision transformer to collaboratively learn knowledge from diverse domains. Concretely, EPVT leverages a set of domain prompts, each of which plays as a domain expert, to capture domain-specific knowledge; and a shared prompt for general knowledge over the entire dataset. To facilitate knowledge sharing and the interaction of different prompts, we introduce a domain prompt generator that enables low-rank multiplicative updates between domain prompts and the shared prompt. A domain mixup strategy is additionally devised to reduce the co-occurring artifacts in each domain, which allows for more flexible decision margins and mitigates the issue of incorrectly assigned domain labels. Experiments on four out-of-distribution datasets and six different biased ISIC datasets demonstrate the superior generalization ability of EPVT in skin lesion recognition across various environments. Code is avaliable at https://github.com/SiyuanYan1/EPVT.
BibTeX
@inproceedings{yan2023epvt,
  author = {Siyuan Yan and Chi Liu and Zhen Yu and Lie Ju and Dwarikanath Mahapatra and Victoria Mar and Monika Janda and H. Peter Soyer and Zongyuan Ge},
  title = {{EPVT: Environment-Aware Prompt Vision Transformer for Domain Generalization in Skin Lesion Recognition}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
}
Figure