TL;DR
This paper compares several deep learning models for automatically counting mpox lesions in patient photographs, a crucial but labor-intensive task. They found that both semantic and instance segmentation methods performed comparably, with no significant improvement from an ensemble of models, suggesting data limitations, not algorithmic choice, are the primary bottleneck.
What Problem Does This Paper Address?
Accurately counting mpox lesions is vital for assessing disease severity and progression, as recommended by the World Health Organization. However, manual counting is time-consuming, prone to human error, and particularly challenging in endemic regions where patients can have thousands of lesions. Previous work utilized a UNet-based semantic segmentation model, but this approach struggles with overlapping or coalescing lesions. This paper aims to address this by evaluating newer semantic and instance segmentation methods, including Mask R-CNN, YOLOv8, and E2EC, against the baseline UNet and a UNet++ model, and to see if an ensemble of these models offers further improvement.
Methods
The researchers evaluated four new models against their existing UNet baseline:
- Mask R-CNN: An instance segmentation network that uses a region proposal network to identify object bounding boxes before segmenting them.
- YOLOv8: A fast, one-stage instance segmentation network that doesn't require a region proposal network.
- E2EC (End-to-End Contour-based): A novel instance segmentation model that represents objects as contours for precise boundaries and efficient processing.
- UNet++: An extension of the UNet model with enhanced connectivity and deep supervision for more accurate semantic segmentation, especially for objects at different scales.
They also created an ensemble model by averaging the predictions of UNet++, Mask R-CNN, and YOLOv8. Performance was assessed using patient-level leave-one-out cross-validation, with metrics including the F1 score for lesion detection and Bland-Altman analysis for lesion count agreement.
Datasets
The study utilized a dataset of 66 mpox photographs from 18 patients, previously collected in the Democratic Republic of the Congo. These photographs were captured using consumer-grade digital cameras under varying conditions, leading to diverse lighting, backgrounds, and imaging distances. Manual, pixel-level segmentation masks and lesion counts were created by a trained rater for ground truth.
Key Insights and Ideas
- Segmentation vs. Instance Segmentation: The study highlights that for this specific task, both semantic segmentation (like UNet) and instance segmentation (like Mask R-CNN, YOLOv8, E2EC) can achieve comparable performance in lesion counting. This suggests that the ability to distinguish individual lesions might not be the primary limiting factor.
- Ensemble Limitations: A surprising finding is that an ensemble of the best-performing models did not significantly outperform the best individual model. This points towards a common failure mode across the models, where errors are frequently shared, rather than being independent.
- Data is King (or Queen): The authors strongly suggest that the performance ceiling is likely dictated by the quality and quantity of the available training data, rather than the sophistication of the deep learning architecture. The variability in photographs and the limited dataset size may be hindering further improvements.
- Lesion Size Matters: Smaller lesions are harder to detect, leading to higher error rates. This is likely due to the limited pixel representation of smaller lesions, making them more susceptible to noise and misclassification.
Key Results
- F1 Scores: The best-performing model was UNet++ with a median F1 score of 0.81, closely followed by the baseline UNet at 0.79. Mask R-CNN and YOLOv8 achieved 0.75, while E2EC scored 0.70. The ensemble model achieved an F1 score of 0.78.
- Lesion Count Agreement (Bland-Altman): UNet++ and Mask R-CNN showed the narrowest limits of agreement (LoA width) at 62.1 and 62.2 respectively, indicating more consistent lesion counts compared to the ground truth. YOLOv8 had the widest LoA at 91.3. The ensemble had an LoA width of 67.4.
- Statistical Significance: T-tests indicated that only YOLOv8 and E2EC had significantly lower F1 scores compared to the baseline UNet. No significant differences were found for LoA width using F-tests for equal variance.
- Undercounting Trend: All models exhibited a trend of undercounting lesions, as indicated by negative mean bias in the Bland-Altman analysis.
Limitations and Future Work
- Dataset Size and Diversity: The study acknowledges the limitation of a small dataset (66 images from 18 patients) and that all patients had the same skin type (Fitzpatrick VI). This limits the generalizability of the models to different skin types, as previous research has shown segmentation models can struggle with underrepresented skin tones.
- Single Rater: Ground truth was provided by a single rater, meaning inter-rater variability could not be assessed.
- Distinguishing Lesion States: The current models cannot distinguish between resolved and unresolved lesions, which is important for clinical assessment.
- Co-infections: The models are not trained to differentiate mpox lesions from similar-looking lesions in co-infections like chickenpox.
Future work suggested includes:
- Exploring advanced hyperparameter tuning, such as optimizing anchor sizes in object detection networks, to improve detection of smaller lesions.
- Investigating the performance across diverse skin types with larger, multi-rater annotated datasets.
- Developing methods to distinguish between resolved and unresolved lesions.
- Investigating the model's ability to handle co-infections.
- Experimenting with newer frameworks like Grounding DINO and SAM2, though initial attempts were not successful.
Summary updated