Skin disease diagnosis with deep learning: a review

Hongfeng Li, Yini Pan, Jie Zhao, Li Zhang

Abstract: Skin cancer is one of the most threatening diseases worldwide. However, diagnosing skin cancer correctly is challenging. Recently, deep learning algorithms have emerged to achieve excellent performance on various tasks. Particularly, they have been applied to the skin disease diagnosis tasks. In this paper, we present a review on deep learning methods and their applications in skin disease diagnosis. We first present a brief introduction to skin diseases and image acquisition methods in dermatology, and list several publicly available skin datasets for training and testing algorithms. Then, we introduce the conception of deep learning and review popular deep learning architectures. Thereafter, popular deep learning frameworks facilitating the implementation of deep learning algorithms and performance evaluation metrics are presented. As an important part of this article, we then review the literature involving deep learning methods for skin disease diagnosis from several aspects according to the specific tasks. Additionally, we discuss the challenges faced in the area and suggest possible future research directions. The major purpose of this article is to provide a conceptual and systematically review of the recent works on skin disease diagnosis with deep learning. Given the popularity of deep learning, there remains great challenges in the area, as well as opportunities that we can explore in the future.

Introduction

Skin cancer represents a significant global health crisis, with melanoma being particularly fatal if not detected in its early stages. Traditionally, manual diagnosis has been highly subjective, with even experienced specialists showing significant variance in their assessments. The core challenge in modern dermatology is building objective, scalable diagnostic tools capable of handling the immense visual variety of skin lesions across diverse populations.

Examples of diverse skin lesions illustrating the complexity of visual diagnosis.

Early Computer-Aided Diagnosis (CAD) systems relied on "handcrafted features"—manually defined rules for color, texture, and shape. These systems often lacked robustness and failed to generalize across different skin types or lighting conditions. The transition to deep learning allows for automated feature learning, which can identify subtle patterns that are difficult for humans to describe mathematically but essential for diagnosis.

Deep Learning Methodologies

This review synthesizes methodologies from hundreds of recent studies, identifying the Convolutional Neural Network (CNN) as the primary technical driver. Key approaches include:

  • Architectural Evolution: The field has transitioned from classic models like AlexNet and VGG to modern, efficient structures like ResNet (utilizing skip connections) and DenseNet.
  • Lesion Segmentation: For isolating lesions, the U-Net and Fully Convolutional Networks (FCNs) are preferred, performing pixel-level classification to create precise boundary masks. Accurate segmentation significantly boosts the accuracy of final classification.
  • Transfer Learning: Given that medical datasets are often small, researchers frequently "pre-train" models on large-scale general datasets (e.g., ImageNet) before "fine-tuning" them on specific dermatological images.
  • Multi-task Learning: Emerging models are designed to perform segmentation and classification simultaneously, leveraging shared representations to improve overall diagnostic performance.

Data Resources

Public datasets are essential for deep learning research. The paper highlights several key repositories:

  • ISIC (International Skin Imaging Collaboration): The primary benchmark, containing over 20,000 images.
  • HAM10000: A large collection of multi-source dermatoscopic images of common pigmented lesions.
  • PH2: A high-quality dataset frequently used for segmentation benchmarking.
  • Dermnet: A vast photo-dermatology source covering 23 disease categories with over 23,000 images.
  • Specialized Datasets: Collections like Asan focus on specific demographics (e.g., Asian skin types), addressing critical gaps in representation.

Key Results and Insights

Landmark studies, such as Esteva et al. (2017), have demonstrated CNNs achieving dermatologist-level performance on clinical images. Quantitative benchmarks show that top-tier models frequently achieve over 90% sensitivity for melanoma detection, while modern segmentation variants achieve Jaccard indices exceeding 0.90 on benchmark datasets like ISIC.

Innovative trends include the use of Generative Adversarial Networks (GANs) to synthesize realistic images of rare diseases, solving class imbalance issues. Furthermore, integrating clinical context—such as patient age, sex, and lesion location—into neural networks is identified as a key path to improving performance beyond pixel analysis alone.

Challenges and Future Directions

Despite technical advancements, several hurdles remain for clinical deployment:

  • Explainability: The "black box" nature of deep learning makes clinicians hesitant. Future research must focus on Explainable AI (XAI) to map model decisions to clinical criteria like the ABCDE rules.
  • Dataset Bias: Current data is heavily skewed toward fair-skinned populations, potentially leading to inaccuracies for patients with darker skin tones.
  • Data Quality: Real-world images often contain artifacts like hair or ruler marks, requiring models to be more robust to noise.
  • Clinical Integration: Moving from high-performing research papers to reliable clinical tools requires rigorous real-world testing and deployment strategies.

Summary updated

Abstract

Skin cancer is one of the most threatening diseases worldwide. However, diagnosing skin cancer correctly is challenging. Recently, deep learning algorithms have emerged to achieve excellent performance on various tasks. Particularly, they have been applied to the skin disease diagnosis tasks. In this paper, we present a review on deep learning methods and their applications in skin disease diagnosis. We first present a brief introduction to skin diseases and image acquisition methods in dermatology, and list several publicly available skin datasets for training and testing algorithms. Then, we introduce the conception of deep learning and review popular deep learning architectures. Thereafter, popular deep learning frameworks facilitating the implementation of deep learning algorithms and performance evaluation metrics are presented. As an important part of this article, we then review the literature involving deep learning methods for skin disease diagnosis from several aspects according to the specific tasks. Additionally, we discuss the challenges faced in the area and suggest possible future research directions. The major purpose of this article is to provide a conceptual and systematically review of the recent works on skin disease diagnosis with deep learning. Given the popularity of deep learning, there remains great challenges in the area, as well as opportunities that we can explore in the future.
BibTeX
@misc{li2020skin,
  author = {Hongfeng Li and Yini Pan and Jie Zhao and Li Zhang},
  title = {{Skin disease diagnosis with deep learning: a review}},
  year = {2020},
  eprint = {2011.05627},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.05627},
}
Figure