Lesion Elevation Prediction from Skin Images Improves Diagnosis

Kumar Abhishek, Ghassan Hamarneh

Abstract

Clinical assessment of skin lesions relies heavily on the "ABCDE" criteria, where "Elevation" (E) traditionally requires physical palpation. In teledermatology and automated 2D image analysis, this critical 3D information is typically lost. This paper presents a novel approach to recover this feature by training deep learning models to predict lesion elevation—categorized as flat, palpable, or nodular—directly from standard RGB images. By integrating these predicted elevation labels as auxiliary features into diagnostic classifiers, the authors demonstrate significant performance improvements, including AUROC increases of up to 6.29% across multiple benchmark datasets. The results suggest that 2D images contain sufficient subtle cues—such as shadows and texture gradients—to serve as a digital proxy for physical touch.

Introduction

In dermatology, the gold standard for identifying potential skin cancers is the "ABCDE" framework: Asymmetry, Border, Color, Diameter, and Evolving/Elevation. While computer-aided diagnosis (CAD) systems have made significant strides in analyzing the first four criteria using 2D images, the "Elevation" component remains a significant hurdle. In a traditional clinical setting, a dermatologist can palpate a lesion to determine if it is raised; however, in the growing field of teledermatology, this tactile information is unavailable.

Existing software solutions for depth estimation, such as those developed for autonomous driving or robotics, generally fail when applied to skin images. These models are typically trained on a scale of meters rather than millimeters and rely on scene assumptions that do not hold for close-up medical photography. This research addresses the "missing E" by investigating whether deep learning can accurately "hallucinate" elevation labels from 2D images and whether these labels can materially improve diagnostic accuracy.

Methodology

The researchers developed a three-stage pipeline to bridge the gap between 2D imagery and 3D clinical features:

  1. Elevation Prediction: Eight different deep learning architectures (including ResNet, MobileNet, and VGG) were trained to classify images into three elevation categories: flat, palpable, or nodular. Among these, VGG-16 emerged as the most effective architecture for capturing the necessary spatial features.
  2. Feature Integration: To incorporate this information into a diagnostic workflow, the researchers modified standard classification models. The one-hot encoded elevation label (either ground truth or predicted) was concatenated with image features at the global average pooling layer before the final classification stage.
  3. Cross-Domain Inference: Because most large-scale skin datasets lack elevation labels, the authors used a model trained on the derm7pt dataset to infer elevation for five external datasets. This allowed them to test whether "inferred" elevation could improve diagnostic performance even in the absence of original ground-truth elevation data.

Datasets and Experimental Setup

The study utilized a diverse range of skin lesion datasets to ensure robust results:

  • derm7pt: The primary training set, containing 1,011 cases with ground-truth labels for three elevation levels.
  • ISIC (2016, 2017, 2018): Large-scale dermoscopic image datasets used to evaluate the diagnostic boost provided by elevation data.
  • MSK & DermoFit: Datasets used to test the generalization of elevation-aware models across different clinical and dermoscopic environments.
  • PAD-UFES-20: A dataset containing binary elevation data, used for additional validation.

Key Insights

2D Images as 3D Proxies

The study's most significant finding is that standard 2D images contain enough subtle cues—specifically shadows, lighting gradients, and texture changes—for a Convolutional Neural Network (CNN) to estimate elevation with approximately 85% accuracy. This suggests that AI can act as a digital substitute for physical examination in remote care settings.

The Failure of General Depth Models

The authors found that off-the-shelf monocular depth models, such as MiDaS, are ineffective for skin lesions. This is attributed to "scene anisotropy." Natural images usually have a ground plane where the bottom of the image is closer to the viewer, a rule that is irrelevant in close-up skin photography. Specialized training on dermatological data is therefore essential.

Generalization Across Domains

Elevation features learned from the derm7pt dataset proved highly transferable to other datasets like ISIC. This indicates that "elevation" is a consistent physical feature that transcends specific camera types or lighting conditions, making it a reliable auxiliary input for CAD systems.

Results

  • Diagnostic Performance: The inclusion of elevation information consistently improved diagnostic accuracy. The authors reported AUROC increases of up to 6.29% for dermoscopic images and 2.69% for clinical images. These improvements were statistically significant ($p < 0.05$) across nearly all tested datasets.
  • Model Interpretability: Using Grad-CAM (Gradient-weighted Class Activation Mapping), the researchers confirmed that the models were focusing on the lesion itself and its immediate boundaries to determine elevation, rather than relying on background artifacts or image shortcuts.
  • Architecture Performance: VGG-16 outperformed more modern architectures like EfficientNet for the specific task of elevation prediction. This is likely due to VGG's architecture being particularly well-suited for processing the local spatial features associated with height and texture.

Limitations and Future Work

While the study proves the value of elevation prediction, several challenges remain. The current model uses three discrete classes (flat, palpable, nodular). Future research should aim for "dense" elevation maps that provide millimeter-scale measurements across the entire lesion surface. Additionally, while the model can identify if a lesion is "raised," it does not yet provide absolute physical height, which is a critical metric for staging cancers like melanoma (e.g., Breslow’s depth). Expanding the training data to include images paired with 3D scanner ground truth could further refine these predictions.

Summary updated

Abstract

While deep learning-based computer-aided diagnosis for skin lesion image analysis is approaching dermatologists' performance levels, there are several works showing that incorporating additional features such as shape priors, texture, color constancy, and illumination further improves the lesion diagnosis performance. In this work, we look at another clinically useful feature, skin lesion elevation, and investigate the feasibility of predicting and leveraging skin lesion elevation labels. Specifically, we use a deep learning model to predict image-level lesion elevation labels from 2D skin lesion images. We test the elevation prediction accuracy on the derm7pt dataset, and use the elevation prediction model to estimate elevation labels for images from five other datasets: ISIC 2016, 2017, and 2018 Challenge datasets, MSK, and DermoFit. We evaluate cross-domain generalization by using these estimated elevation labels as auxiliary inputs to diagnosis models, and show that these improve the classification performance, with AUROC improvements of up to 6.29% and 2.69% for dermoscopic and clinical images, respectively. The code is publicly available at https://github.com/sfu-mial/LesionElevation.
BibTeX
@inproceedings{abhishek2024lesion,
  author = {Kumar Abhishek and Ghassan Hamarneh},
  title = {{Lesion Elevation Prediction from Skin Images Improves Diagnosis}},
  booktitle = {ISIC/iMIMIC/EARTH/DeCaF @ MICCAI},
  year = {2024},
  doi = {10.1007/978-3-031-77610-6\_5},
}
Figure