A Novel Multi-task Model Imitating Dermatologists for Accurate Differential Diagnosis of Skin Diseases in Clinical Images

Yan-Jie Zhou, Wei Liu, Yuan Gao, Jing Xu, Le Lu, Yuping Duan, Hao Cheng, Na Jin, Xiaoyong Man, Shuang Zhao, Yu Wang

Introduction

Automated skin disease diagnosis has traditionally relied heavily on dermoscopy—specialized magnified imaging that eliminates surface reflections. However, in most real-world clinical settings, initial assessments are made using standard cameras or smartphones. These "clinical images" introduce significant hurdles for computer vision models, including cluttered backgrounds (clothing, hair, furniture), high intra-class variability, and low inter-class variability where different diseases present similar visual characteristics. Furthermore, most existing AI models function as "black boxes," providing a diagnosis without the contextual reasoning—such as lesion location or specific morphological attributes—that human dermatologists use to differentiate between similar conditions.

To address these challenges, this research introduces DermImitFormer, a Vision Transformer (ViT)-based framework designed to mimic the diagnostic logic of medical professionals. By simultaneously identifying the disease, the affected body part, and specific lesion attributes, the model provides a more robust and interpretable diagnostic tool for clinical photography.

Methodology: Mimicking the Dermatologist's Workflow

DermImitFormer utilizes a Multi-Task Learning (MTL) approach, moving beyond simple classification to a holistic understanding of the patient's condition. The architecture is built upon three specialized components:

1. Multi-Task Learning (MTL)

The model features three distinct "heads" that predict the disease label, the body part (e.g., face, arm, trunk), and specific lesion attributes (e.g., plaque, scale, crust). This structure forces the model to learn the underlying correlations between these factors, mirroring how a clinician considers the context of a lesion during a physical exam.

2. Lesion Selection Module (LSM)

To combat the noise inherent in clinical photos, the LSM acts as a "zoom-in" mechanism. Leveraging the self-attention scores within the Vision Transformer, the module identifies and selects the top-$K$ most informative image patches (tokens) related to the lesion. By mathematically "cropping" the image to focus on relevant features, the model effectively discards background clutter like clothing or furniture without requiring expensive manual bounding box annotations.

3. Cross-Interaction Module (CIM)

The CIM serves as the model's reasoning engine. Using cross-attention, it fuses features from the disease, body part, and attribute tasks. This is crucial for resolving visual ambiguity. For instance, while rosacea and tinea versicolor may share similar textures (macules or papules), rosacea predominantly affects the face. The CIM allows the model to use the body part context to refine its disease prediction, resolving ambiguities that might confuse a single-task model.

Data and Benchmarking

The effectiveness of DermImitFormer was evaluated using three datasets, including a significant new contribution to the field:

  • Derm-49: A large-scale clinical dataset established by the authors, containing 57,246 images across 49 common diseases, 15 body parts, and 27 lesion attributes. This dataset represents a major step forward in clinical skin disease benchmarking due to its size and diversity.
  • SD-198: A public benchmark consisting of 6,584 images across 198 categories.
  • PAD-UFES-20: A dataset featuring 2,298 samples of six skin diseases, including detailed metadata like body part location.

Experimental Results

DermImitFormer demonstrated superior performance over existing state-of-the-art (SOTA) models across all benchmarks:

  • Derm-49: The model achieved 82.6% accuracy, outperforming standard ViT and Swin Transformer baselines by more than 2%.
  • SD-198: It reached 74.5% accuracy, a notable improvement over previous methods such as Dual Stream (71.4%) and SPBL (67.8%).
  • Ablation Studies: The importance of the individual modules was confirmed through ablation testing. Removing the LSM (zoom-in feature) resulted in a 1.6% drop in accuracy, while removing the CIM (reasoning module) decreased the disease F1-score from 79.5% to 79.1%, proving that multi-task interaction is essential for high-performance diagnosis.

Conclusion and Future Work

DermImitFormer represents a significant advancement in clinical skin disease diagnosis by integrating contextual reasoning into a deep learning framework. By focusing on relevant lesion tokens and fusing multi-task information, the model achieves high accuracy and provides a more interpretable output that aligns with dermatological practice.

Future research will focus on making the lesion selection process adaptive, as the current fixed number of tokens ($K$) may not be optimal for lesions of varying sizes. Additionally, the authors aim to investigate the computational efficiency of the model for mobile deployment and conduct user studies to determine how clinicians perceive and trust the model's attribute-based reasoning in live diagnostic environments.

Summary updated

BibTeX
@inproceedings{zhou2023novel,
  author = {Yan-Jie Zhou and Wei Liu and Yuan Gao and Jing Xu and Le Lu and Yuping Duan and Hao Cheng and Na Jin and Xiaoyong Man and Shuang Zhao and Yu Wang},
  title = {{A Novel Multi-task Model Imitating Dermatologists for Accurate Differential Diagnosis of Skin Diseases in Clinical Images}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  doi = {10.1007/978-3-031-43987-2\_20},
}
Figure