Optimizing Skin Lesion Classification via Multimodal Data and Auxiliary Task Integration

Mahapara Khurshid, Mayank Vatsa, Richa Singh

Abstract

Skin cancer detection using smartphone images is hindered by low inter-class variation and high intra-class noise. This paper presents a multimodal deep learning framework that integrates patient metadata with visual features. A novel auxiliary super-resolution task is employed to force the model to learn granular visual details, acting as a powerful regularizer. Evaluated on the PAD-UFES20 dataset, the proposed method achieves a balanced accuracy of 0.832 and an AUC of 0.960, significantly surpassing existing state-of-the-art approaches.

Introduction

Skin cancer is the most prevalent cancer globally, making early detection the single most important factor in survival. While specialized dermoscopy tools are standard in clinical settings, accessibility remains a major barrier in under-resourced areas. Smartphone-based screening offers a scalable solution but faces two primary difficulties: different diseases often look nearly identical (low inter-class variation), and the same disease can vary significantly depending on lighting, skin tone, and camera quality (high intra-class variation).

The authors address these challenges by mimicking a dermatologist’s diagnostic process: combining visual evidence with the patient's clinical history. The study utilizes the PAD-UFES20 dataset, which contains 2,298 real-world smartphone images across six categories, including Melanoma (MEL), Basal Cell Carcinoma (BCC), and Nevus (NEV).

Methodology

The proposed architecture integrates visual and textual data through three main pillars:

  1. Multimodal Feature Extraction: Visual features are extracted using standard CNN backbones (e.g., VGG13, ResNet50) after images are normalized via Gray Color Constancy and CLAHE. Simultaneously, a four-layer fully connected network processes 21 clinical metadata features, such as patient age, lesion location, and whether the lesion has grown or bled.
  2. Auxiliary Super-Resolution Task: This is the framework's "secret sauce." Alongside classification, the model is tasked with predicting a high-resolution version of the input image. Because the branches share an encoder, the model is forced to extract much finer, more granular visual features than it would through classification alone.
  3. Feature-level Fusion: Instead of simple concatenation, the authors use a multiplication-based fusion where metadata features "weigh" the visual features, highlighting the most relevant parts of the image based on the patient's history.
Proposed multimodal architecture featuring the auxiliary super-resolution branch.

The final training loss combines Weighted Cross-Entropy (to handle class imbalance) and Mean Squared Error for the super-resolution task.

Key Results and Insights

The proposed method consistently outperformed existing state-of-the-art (SOTA) approaches across multiple architectures. VGG13 emerged as the top-performing backbone for this task.

  • Performance Metrics: The model achieved a Balanced Accuracy (BACC) of 0.832, surpassing the previous SOTA of 0.806. It reached a peak Area Under the Curve (AUC) of 0.960.
  • Reliability: The model maintained a specificity of over 90% for all classes, which is critical in medical settings to avoid unnecessary biopsies and patient anxiety.
  • Clustering Quality: t-SNE visualizations confirmed that the proposed model created much tighter, more distinct clusters for each disease compared to previous methods.
Comparative performance metrics across different architectures and metadata integration strategies.

Discussion and Future Work

A key takeaway is the role of super-resolution as a regularizer; it ensures the neural network doesn't ignore tiny textures that differentiate malignant lesions from benign ones. Metadata acts as a dynamic filter, up-weighting specific visual triggers associated with high-risk clinical profiles.

Despite these gains, the model still struggles to distinguish between Basal Cell Carcinoma (BCC) and Squamous Cell Carcinoma (SCC), as these often require a physical biopsy for definitive identification even by human experts. Future directions include integrating other imaging modalities, such as X-rays and MRIs, and expanding auxiliary tasks to include lesion segmentation.

Summary updated

Abstract

The rising global prevalence of skin conditions, some of which can escalate to life-threatening stages if not timely diagnosed and treated, presents a significant healthcare challenge. This issue is particularly acute in remote areas where limited access to healthcare often results in delayed treatment, allowing skin diseases to advance to more critical stages. One of the primary challenges in diagnosing skin diseases is their low inter-class variations, as many exhibit similar visual characteristics, making accurate classification challenging. This research introduces a novel multimodal method for classifying skin lesions, integrating smartphone-captured images with essential clinical and demographic information. This approach mimics the diagnostic process employed by medical professionals. A distinctive aspect of this method is the integration of an auxiliary task focused on super-resolution image prediction. This component plays a crucial role in refining visual details and enhancing feature extraction, leading to improved differentiation between classes and, consequently, elevating the overall effectiveness of the model. The experimental evaluations have been conducted using the PAD-UFES20 dataset, applying various deep-learning architectures. The results of these experiments not only demonstrate the effectiveness of the proposed method but also its potential applicability under-resourced healthcare environments.
BibTeX
@inproceedings{khurshid2024optimizing,
  author = {Mahapara Khurshid and Mayank Vatsa and Richa Singh},
  title = {{Optimizing Skin Lesion Classification via Multimodal Data and Auxiliary Task Integration}},
  booktitle = {ISBI},
  year = {2024},
  url = {https://openreview.net/forum?id=ii2zF6qKOa},
}
Figure