EPVT: Environment-aware Prompt Vision Transformer for Domain Generalization in Skin Lesion Recognition

Siyuan Yan, Chi Liu, Zhen Yu, Lie Ju, Dwarikanath Mahapatrainst, Victoria Mar, Monika Janda, Peter Soyer, Zongyuan Ge

TL;DR

EPVT (Environment-aware Prompt Vision Transformer) combats the tendency of skin lesion classifiers to overfit to disease-irrelevant image artifacts like hairs, rulers, or dark corners. By combining domain-specific and shared prompts via a low-rank prompt generator, EPVT effectively disentangles clinical features from environmental noise. This approach achieves state-of-the-art domain generalization across multiple unseen clinical and dermoscopic datasets.

What Problem Does This Paper Address?

Deep learning models have shown immense promise in computer-aided diagnosis (CAD) for skin cancer. However, these models are notorious for exploiting "spurious cues"—disease-irrelevant artifacts frequently present in medical datasets, such as calibration rulers, hair, gel bubbles, and dark camera corners.

When a model overfits to these artifacts (e.g., associating the presence of a ruler with a higher probability of malignancy because doctors tend to measure larger tumors), it fails dramatically when deployed in real-world environments where these artifacts are absent or altered.

Learning domain-invariant features in medical imaging is exceptionally difficult due to the immense diversity in lesion shapes, colors, and textures. Previous domain generalization (DG) methods often relied on weak domain experts (like batch normalization statistics) or failed to facilitate collaborative knowledge sharing between domains, leaving models vulnerable to distribution shifts.

Fig. 1. The training data is split into five domains: clean, rulers, hairs, air pockets, and dark corners. Domain generalization aims to train the model to learn from these domains to generalize well in unseen domains.

Methods

The authors introduce EPVT, an environment-aware prompt vision transformer that treats different types of image artifacts as distinct "source domains."

The architecture consists of three core components:

  1. Domain-specific Prompt Learning: Built on top of a Vision Transformer (ViT) backbone, EPVT prepends learnable prompt tokens to the input sequence. Each domain (defined by a specific artifact type) gets its own prompt, forcing it to act as a specialized "domain expert."
  2. Cross-domain Knowledge Learning (Domain Prompt Generator): To prevent domain prompts from learning completely isolated representations, the authors introduce a low-rank prompt generator. Each domain prompt ($P^m$) is decomposed into a Hadamard (element-wise) product of a globally shared prompt ($P^*$) and a domain-specific rank-one matrix ($P_k = u_k v_k^T$). This structure encourages cross-domain information sharing while preserving unique domain characteristics.
  3. Domain Mixup Strategy: Real-world images often contain multiple overlapping artifacts (e.g., both hair and a dark corner). EPVT employs a domain mixup strategy during training, blending images from different domains and applying soft cross-entropy losses to smooth out decision margins and handle ambiguous domain labels.

During inference, a lightweight Adapter module dynamically estimates the linear correlation between the target image's features and the source domain prompts, combining them into an optimized, adapted prompt for final prediction.

Fig. 2. The overview of our environment-aware prompt vision transformer (EPVT).

Datasets

The model was trained and evaluated using several prominent skin lesion benchmarks:

  • Source/Training Dataset: ISIC2019, partitioned into five distinct domains based on artifact annotations: dark corner (2,351 images), hair (4,884 images), gel bubble (1,640 images), ruler (672 images), and clean (2,796 images).
  • Out-of-Distribution (OOD) Target Datasets:
  • Derm7pt-Dermoscopic & Derm7pt-Clinical (evaluating shifts between dermoscopic and standard smartphone clinical images).
  • PH2 (dermoscopic images).
  • PAD-UFES-20 (clinical smartphone images).
  • Trap Sets: Six synthetic splits derived from ISIC2019 with artificially injected bias levels ranging from 0 (no bias) to 1 (maximum adversarial bias, where artifacts strongly correlate with the wrong class label).

Key Insights and Ideas

  • Artifacts as Domains: Instead of attempting to pre-process and remove artifacts (which can inadvertently destroy clinical information), EPVT explicitly embraces them as domain labels. This allows the transformer's self-attention mechanism to model the relationship between artifact tokens and image patches, effectively learning what to ignore.
  • Low-Rank Multiplicative Updates: Borrowing insights from parameter-efficient fine-tuning in NLP, the authors show that domain-specific updates possess a low intrinsic rank. Decomposing prompts into rank-one subspaces ensures efficient parameter usage and prevents overfitting.
  • Intelligent Prompt Weighting: The adapter successfully learns a negative correlation between domain distance and prompt weights. For example, when evaluating on the Derm7pt-Clinical dataset, the model automatically assigns the highest weight to the "dark corner" domain prompt because it shares the closest feature distribution, validating that the adapter behaves logically.

Key Results

EPVT demonstrates superior robustness and generalization performance across the board:

  • OOD Generalization: As shown in the results table below, EPVT achieved the highest average ROC-AUC of 84.11% across the four unseen target datasets, outperforming standard Empirical Risk Minimization (ERM) by 4.48% and beating the previous best DG method (DoPrompt) by 2.05%.
  • Debiasing Performance: On the most severe adversarial "trap set" (bias degree 1), standard ERM performance collapsed to 65.2% AUC due to its reliance on spurious cues. EPVT maintained a robust 74.6% AUC, outperforming the baseline by 9.4%.
Table 1. The comparison on out-of-distribution datasets

Limitations and Future Work

The primary limitation of EPVT is its reliance on explicit artifact annotations to define source domains during the training phase. Manually labeling thousands of images for dark corners, hairs, and rulers is labor-intensive and may not scale to all medical imaging modalities.

The authors suggest that future research should focus on unsupervised or self-supervised discovery of artifact-based domains to make the pipeline fully automated.

Summary updated

Abstract

Skin lesion recognition using deep learning has made remarkable progress, and there is an increasing need for deploying these systems in real-world scenarios. However, recent research has revealed that deep neural networks for skin lesion recognition may overly depend on disease-irrelevant image artifacts (i.e., dark corners, dense hairs), leading to poor generalization in unseen environments. To address this issue, we propose a novel domain generalization method called EPVT, which involves embedding prompts into the vision transformer to collaboratively learn knowledge from diverse domains. Concretely, EPVT leverages a set of domain prompts, each of which plays as a domain expert, to capture domain-specific knowledge; and a shared prompt for general knowledge over the entire dataset. To facilitate knowledge sharing and the interaction of different prompts, we introduce a domain prompt generator that enables low-rank multiplicative updates between domain prompts and the shared prompt. A domain mixup strategy is additionally devised to reduce the co-occurring artifacts in each domain, which allows for more flexible decision margins and mitigates the issue of incorrectly assigned domain labels. Experiments on four out-of-distribution datasets and six different biased ISIC datasets demonstrate the superior generalization ability of EPVT in skin lesion recognition across various environments. Code is avaliable at https://github.com/SiyuanYan1/EPVT.
BibTeX
@inproceedings{yan2023epvt,
  author = {Siyuan Yan and Chi Liu and Zhen Yu and Lie Ju and Dwarikanath Mahapatrainst and Victoria Mar and Monika Janda and Peter Soyer and Zongyuan Ge},
  title = {{EPVT: Environment-aware Prompt Vision Transformer for Domain Generalization in Skin Lesion Recognition}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
}
Figure