Joint Dermatological Lesion Classification and Confidence Modeling with Uncertainty Estimation

Gun-Hee Lee, Han-Bin Ko, Seong-Whan Lee

Abstract

Deep learning has played a major role in the interpretation of dermoscopic images for detecting skin defects and abnormalities. However, current deep learning solutions for dermatological lesion analysis are typically limited in providing probabilistic predictions which highlights the importance of concerning uncertainties. This concept of uncertainty can provide a confidence level for each feature which prevents overconfident predictions with poor generalization on unseen data. In this paper, we propose an overall framework that jointly considers dermatological classification and uncertainty estimation together. The estimated confidence of each feature to avoid uncertain feature and undesirable shift, which are caused by environmental difference of input image, in the latent space is pooled from confidence network. Our qualitative results show that modeling uncertainties not only helps to quantify model confidence for each prediction but also helps classification layers to focus on confident features, therefore, improving the accuracy for dermatological lesion classification. We demonstrate the potential of the proposed approach in two state-of-the-art dermoscopic datasets (ISIC 2018 and ISIC 2019).

Introduction

Deep learning has become incredibly effective at identifying skin cancer from dermoscopic images, sometimes rivaling expert dermatologists. However, standard models suffer from two major flaws in a clinical setting: overconfidence and a lack of nuance. Standard models often produce high-probability predictions even when the input is noisy or contains "distractors" like hair, skin folds, or medical markers. Furthermore, they represent an image as a single "point" in a high-dimensional space; if that point is shifted by noise, the classification fails.

In medicine, a clinician provides a level of confidence alongside a diagnosis. If a specific feature of a mole is blurry, they ignore it and look at the clearer parts. This research aims to equip AI models with that same ability to "know what they don't know" by treating features as probabilistic distributions rather than fixed points.

Methods: Probabilistic Latent Representations

The proposed framework moves away from deterministic point vectors to Probabilistic Latent Representations. The architecture consists of a standard "backbone" network (such as EfficientNet or ResNet) to extract features (the mean, $\mu$) and a secondary Uncertainty Estimation Network that predicts the variance ($\sigma^2$) for every feature.

Overview of the proposed framework featuring backbone selection and the uncertainty estimation network for confidence pooling.

The core innovation is Confidence Pooling. Instead of treating all extracted features equally, the model calculates a confidence value ($1/\sigma^2$) for each. It then weights the features so the classifier "listens" to the confident indicators and ignores the uncertain ones. Training occurs in two stages: first, the backbone is trained to classify lesions using standard cross-entropy; second, the backbone is frozen while the uncertainty network is trained using a specialized log-likelihood loss that encourages images of the same class to have overlapping Gaussian distributions in the latent space.

Experimental Results

The model was evaluated on two industry-standard benchmarks: ISIC 2018 (HAM10000) and ISIC 2019. These datasets are notoriously difficult because they are highly imbalanced, with common benign conditions outnumbering rare malignancies by thousands of images.

Performance comparison of various backbone models with and without Confidence Pooling (CP) on the ISIC dataset.

As shown in the results table, adding Confidence Pooling (CP) consistently improved performance across all tested backbones (ResNet, DenseNet, EfficientNet). For instance, using EfficientNet-B4 with CP improved Balanced Accuracy (BACC) from 74.2% to 78.7%. Qualitative results further showed that the model correctly identified images with hair or poor lighting as "high uncertainty," yet remained able to classify them correctly by focusing on the remaining clear features.

Beyond raw accuracy, the framework allows for a "rejection" strategy. In a clinical setting, high-uncertainty images can be automatically flagged for human review. By discarding the bottom 20% of low-confidence samples on the ISIC 2019 dataset, the Balanced Accuracy jumped from 75.5% to 86.7%, demonstrating the system's utility as a reliable diagnostic aid.

Conclusion

Modeling uncertainty is more than a safety feature; it is a performance booster. By penalizing uncertain features during the decision-making process, the model becomes robust to the "noise" common in medical imaging. While the current two-stage training approach is effective, future work may explore end-to-end joint training and more precise clinical thresholds for the rejection metric to balance diagnostic safety with specialist workload.

Summary updated

Abstract

Deep learning has played a major role in the interpretation of dermoscopic images for detecting skin defects and abnormalities. However, current deep learning solutions for dermatological lesion analysis are typically limited in providing probabilistic predictions which highlights the importance of concerning uncertainties. This concept of uncertainty can provide a confidence level for each feature which prevents overconfident predictions with poor generalization on unseen data. In this paper, we propose an overall framework that jointly considers dermatological classification and uncertainty estimation together. The estimated confidence of each feature to avoid uncertain feature and undesirable shift, which are caused by environmental difference of input image, in the latent space is pooled from confidence network. Our qualitative results show that modeling uncertainties not only helps to quantify model confidence for each prediction but also helps classification layers to focus on confident features, therefore, improving the accuracy for dermatological lesion classification. We demonstrate the potential of the proposed approach in two state-of-the-art dermoscopic datasets (ISIC 2018 and ISIC 2019).
BibTeX
@inproceedings{lee2021joint,
  author = {Gun-Hee Lee and Han-Bin Ko and Seong-Whan Lee},
  title = {{Joint Dermatological Lesion Classification and Confidence Modeling with Uncertainty Estimation}},
  booktitle = {ACPR},
  year = {2021},
  url = {https://openreview.net/forum?id=Nz7ORH0y5ve},
}
Figure