TL;DR
This paper investigates how to improve skin lesion classification by combining image data with patient metadata (age, gender, body location). They found that while metadata can boost overall accuracy, it sometimes comes at the cost of reduced sensitivity for less common lesion types, and that using internal CNN features for fusion yields better results than just using final predictions.
What Problem Does This Paper Address?
Early detection of skin cancer significantly improves survival rates. While Convolutional Neural Networks (CNNs) have shown great promise in classifying skin lesions from images, dermatologists in practice don't rely solely on visual information. They also consider patient metadata like age, gender, and the location of the lesion on the body. This paper addresses the gap in understanding how effectively this patient metadata can be integrated with CNN-based image analysis to improve diagnostic accuracy, especially in a way that mimics real-world clinical scenarios. The ISIC 2019 challenge, which included a task specifically for metadata fusion, saw surprisingly mixed results, with some teams even experiencing performance decreases, highlighting the need for a systematic study on effective fusion strategies.
Methods
The authors explore two primary fusion methods:
- Classical Machine Learning (cML) Fusion: This approach concatenates the output of a pre-trained CNN (either its final class probabilities or internal layer activations) with the processed patient metadata. This combined feature vector is then fed into traditional machine learning algorithms like Support Vector Machines (SVM), Gradient Boosting, and Random Forests.
- Shallow Neural Network (sNN) Fusion: Similar to cML fusion, this method concatenates CNN features and metadata. However, instead of traditional ML algorithms, the combined vector is passed through a shallow neural network (2-3 dense layers) followed by a softmax output for classification.
They experimented with two established CNN architectures as baselines: VGG16 and RESNET50. Metadata preprocessing involved normalizing age to a [0,1] range and one-hot encoding categorical features (gender and location) to create a unified metadata vector.
Datasets
The study utilizes the ISIC 2019 skin lesion classification challenge dataset. For their experiments, they used a subset of 22,480 images from the ISIC 2019 training set where all metadata (age, gender, body location) were present. This subset was then split into training, validation, and testing sets, ensuring the same class proportion across all splits to handle the dataset's inherent class imbalance.
Key Insights and Ideas
- Metadata's Double-Edged Sword: While metadata can increase overall classification accuracy, it can sometimes negatively impact sensitivity for less represented classes. This suggests that metadata might inadvertently bias models towards more common lesion types.
- Internal CNN Features Shine: Merging metadata with internal activation layers of the CNN (specifically, the first fully connected layer's output) yielded better results than using just the final softmax predictions. This indicates that deeper, more abstract image features are more compatible with metadata.
- Age is King (for Accuracy): Among the metadata features, age consistently proved to be the most influential for improving classification accuracy, followed by location and then gender.
- Shallow Networks Outperform Classical ML for Fusion: When fusing with metadata, shallow neural networks generally outperformed classical ML algorithms like SVM and Random Forests in terms of performance gains.
- Suboptimal Fusion Strategies: The authors suggest that some teams in the ISIC 2019 challenge might have employed suboptimal strategies for metadata fusion, contributing to the mixed results observed.
Key Results
- Accuracy Gains: The best configurations, particularly those using shallow neural networks with internal CNN features, achieved significant accuracy increases. For example, with VGG16, accuracy improved by up to 19.1% using sNN fusion.
- Sensitivity Trade-offs: Despite accuracy gains, the average sensitivity often decreased. For instance, in one VGG16+sNN configuration, accuracy increased by 19.1%, but sensitivity saw a slight decrease.
- Best Performing Fusion: The combination of internal CNN features (from the first fully connected layer) with metadata, processed by a shallow neural network, consistently yielded the best results across both VGG16 and RESNET50 baselines.
- Metadata Contribution: The combination of "Age + Location" often provided the best performance among metadata combinations, closely followed by "All" metadata.
Limitations and Future Work
- Class Imbalance and Sensitivity: The paper highlights the persistent challenge of class imbalance and its impact on sensitivity. The authors hypothesize that metadata might be more correlated with common classes, thus potentially hindering the detection of rarer ones.
- Optimization Goal: The models were optimized for accuracy, which might explain the decrease in sensitivity. Future work could involve training networks to directly maximize average sensitivity.
- Generalizability: The authors acknowledge that the findings might not directly generalize to other medical contexts or different types of metadata. Further validation in diverse scenarios is needed.
- "Flatten+Metadata" Configuration: Concatenating metadata directly with flattened image features (from early CNN layers) did not yield improvements, likely due to the vast difference in feature vector sizes. This approach was left for future work, possibly requiring more sophisticated size-dependent normalization techniques.
Summary updated