Using Multiple Dermoscopic Photographs of One Lesion Improves Melanoma Classification via Deep Learning: A Prognostic Diagnostic Accuracy Study

Achim Hekler, Roman C. Maron, Sarah Haggenmüller, Max Schmitt, Christoph Wies, Jochen S. Utikal, Friedegund Meier, Sarah Hobelsberger, Frank Friedrich Gellrich, Mildred Sergon, Axel Hauschild, Lars E. French, Lucie Heinzerling, Justin G. Schlager, Kamran Ghoreschi, Max Schlaak, Franz J. Hilke, Gabriela Poch, Sören Korsing, Carola Berking, Markus V. Heppt, Michael Erdmann, Sebastian Haferkamp, Konstantin Drexler, Dirk Schadendorf, Wiebke Sondermann, Matthias Goebeler, Bastian Schilling, Jakob Nikolas Kather, Eva Krieghoff-Henning, Titus J. Brinker

Abstract

Deep learning models for melanoma detection often struggle with robustness and calibration in real-world clinical settings. This study investigates whether using multiple real-world dermoscopic photographs of a single lesion (MV-Real) improves performance compared to single-view or standard test-time augmentation (MV-Artificial) approaches. Evaluating a ConvNeXT-based classifier on a prospective dataset of 656 lesions, we find that the MV-Real approach significantly enhances AUROC, reduces calibration error, and improves robustness. These findings suggest that genuine visual diversity from multiple physical perspectives is superior to digital transformations for enhancing the reliability of AI in dermatology.

Introduction

Convolutional Neural Networks (CNNs) have reached a milestone in dermatology, often matching or exceeding the diagnostic performance of expert clinicians in controlled environments. Despite this success, translating these models to the clinic reveals significant hurdles: brittleness and overconfidence. A model's prediction may fluctuate based on minor variations in image acquisition, such as angle or lighting, indicating a lack of robustness. Furthermore, these networks are often "confident but wrong," yielding high-probability scores for incorrect classifications, which points to poor calibration.

Standard practice often involves "Test-Time Augmentation" (TTA), where a single image is digitally manipulated (e.g., rotated or brightened) to give the model multiple "looks." However, this paper argues that TTA is a mathematical proxy that does not add new information. This study explores whether taking multiple actual photos from different perspectives can solve these reliability issues.

Methods

The researchers employed a state-of-the-art ConvNeXT architecture trained on a large-scale dataset of 29,562 images from the ISIC archive (HAM10000 and BCN20000). To evaluate the impact of multiple views, they utilized the SCP2 dataset, a prospective multicenter collection of 656 suspected melanoma lesions from eight German university hospitals. Each lesion in SCP2 was documented with six distinct real-world photographs, varying in angle, position, and dermatoscope mode (polarized vs. non-polarized).

The study compared three primary diagnostic workflows:

  1. Single-View: A baseline using only the primary "original" image.
  2. MV-Artificial (TTA): Generating five additional views from the original image using digital transforms like rotation, zooming, and brightness adjustments.
  3. MV-Real: Utilizing the six distinct physical photographs taken by the physician.
Comparison of MV-Real and MV-Artificial (TTA) approaches for melanoma classification.

In the multiview approaches, individual predictions for each image were averaged to produce a final diagnostic score. Performance was measured using AUROC for accuracy, Expected Calibration Error (ECE) for probability reliability, and Maximum Confidence Change (MCC) for robustness.

Key Results

The "MV-Real" approach demonstrated superior performance across all evaluated metrics. The diagnostic accuracy (AUROC) improved from 0.905 in the single-view baseline to 0.930. More importantly, the model's reliability saw significant gains: the ECE dropped from 0.131 to 0.072, indicating that the model's confidence scores more accurately reflected the true likelihood of disease. Robustness also improved, with the MCC decreasing from 0.149 to 0.115.

A critical finding was the "Robustness Paradox": while MV-Artificial (TTA) provided a minor boost to accuracy, it actually decreased the model's robustness compared to the single-view baseline. This suggests that digital transformations can introduce noise that makes model predictions more erratic, whereas real-world photographs provide complementary information—such as different perspectives on lesion borders or the mitigation of hair occlusions—that stabilizes the model.

Discussion and Limitations

The results underscore that "more is better" only when the additional data provides genuinely new visual information. The synergy between polarized and non-polarized imaging modes in the MV-Real set likely contributed to this performance leap, as different modes highlight different pathological features.

However, the practical application of this method faces challenges regarding clinical workflow efficiency. Requiring six photos per lesion increases the time burden on dermatologists. Future research should focus on determining the optimal number of images—perhaps two or three—needed to capture the majority of the benefits without significantly slowing down examinations. Additionally, while this study focused on binary classification (melanoma vs. nevus), extending this multiview approach to complex, multiclass diagnostic environments remains a vital next step for clinical integration.

Summary updated

Abstract

Background: Convolutional neural network (CNN)-based melanoma classifiers face several challenges that limit their usefulness in clinical practice.Objective: To investigate the impact of multiple real-world dermoscopic views of a single lesion of interest on a CNN-based melanoma classifier.Methods: This study evaluated 656 suspected melanoma lesions.Classifier performance was measured using area under the receiver operating characteristic curve (AUROC), expected calibration error (ECE) and maximum confidence change (MCC) for (I) a single-view scenario, (II) a multiview scenario using multiple artificially modified images per lesion and (III) a multiview scenario with multiple real-world images per lesion. Results:The multiview approach with real-world images significantly increased the AUROC from 0.905 (95% CI, 0.879-0.929) in the single-view approach to 0.930 (95% CI, 0.909-0.951).ECE and MCC also improved significantly from 0.131 (95% CI, 0.105-0.159)to 0.072 (95% CI: 0.052-0.093)and from 0.149 (95% CI, 0.125-0.171)to 0.115 (95% CI: 0.099-0.131),respectively.Comparing multiview real-world to artificially modified images showed comparable diagnostic accuracy and uncertainty estimation, but significantly worse robustness for the latter. Conclusion:Using multiple real-world images is an inexpensive method to positively impact the performance of a CNN-based melanoma classifier.
BibTeX
@article{hekler2024using,
  author = {Achim Hekler and Roman C. Maron and Sarah Haggenmüller and Max Schmitt and Christoph Wies and Jochen S. Utikal and Friedegund Meier and Sarah Hobelsberger and Frank Friedrich Gellrich and Mildred Sergon and Axel Hauschild and Lars E. French and Lucie Heinzerling and Justin G. Schlager and Kamran Ghoreschi and Max Schlaak and Franz J. Hilke and Gabriela Poch and Sören Korsing and Carola Berking and Markus V. Heppt and Michael Erdmann and Sebastian Haferkamp and Konstantin Drexler and Dirk Schadendorf and Wiebke Sondermann and Matthias Goebeler and Bastian Schilling and Jakob Nikolas Kather and Eva Krieghoff-Henning and Titus J. Brinker},
  title = {{Using Multiple Dermoscopic Photographs of One Lesion Improves Melanoma Classification via Deep Learning: A Prognostic Diagnostic Accuracy Study}},
  journal = {Journal of the American Academy of Dermatology},
  year = {2024},
  doi = {10.1016/j.jaad.2023.11.065},
}
Figure