Out-of-Distribution Detection for Long-tailed and Fine-grained Skin Lesion Images

Deval Mehta, Yaniv Gal, Adrian Bowling, C. Paul Bonnington, Zongyuan Ge

Abstract

Recent years have witnessed a rapid development of automated methods for skin lesion diagnosis and classification. However, as these systems move toward clinical deployment, robustness against Out-of-Distribution (OOD) samples—unknown skin conditions or non-medical images—remains a critical challenge. Standard deep learning models often misclassify OOD samples into known categories with high confidence. This paper proposes a strategic approach to improve OOD detection by addressing the long-tailed and fine-grained nature of clinical skin lesion datasets. By combining targeted Mixup data augmentation for middle and tail classes with prototype learning, we enhance the model's ability to define clear decision boundaries. Our experiments on large-scale datasets demonstrate significant improvements in OOD detection performance while maintaining high classification accuracy for known categories.

Introduction

Deep learning has revolutionized dermatology, yet a significant hurdle remains: the "confidence problem." In a clinical setting, a model must not only classify known diseases accurately but also recognize when it is presented with a condition it has never seen before. Current models tend to force-map these unknown (OOD) samples into existing categories, potentially leading to dangerous misdiagnoses.

While OOD detection is a well-studied field, most research relies on balanced, coarse-grained datasets like CIFAR-10. Real-world medical data is far more complex, characterized by two primary challenges:

  1. Long-tailed Distribution: A few common conditions (head classes) dominate the dataset, while many rare diseases (tail classes) have very few examples.
  2. Fine-grained Similarity: Different skin lesions often share subtle visual features, making the boundaries between classes—and the boundary for "unknown" samples—extremely blurry.

Methodology

To address these challenges, we propose a two-pronged strategy designed to sharpen decision boundaries and level the playing field for rare classes.

Targeted Mixup (M-T Intersubset)

Standard Mixup augmentation blends images and labels to teach models about the interpolative space between categories. However, in long-tailed datasets, standard Mixup is dominated by common "head" classes. We introduce a targeted strategy that partitions the data into Head, Middle, and Tail subsets. Our approach focuses Mixup specifically between the Middle and Tail classes (the MX5 strategy). This forces the feature extractor to learn more discriminative representations for rare conditions, which are the most likely to be confused with OOD samples.

Prototype Learning Integration

To handle the fine-grained nature of skin lesions, we integrate Mixup with Prototype Learning. In this framework, the model learns a central "prototype" (a representative point in the feature space) for every class. We utilize a custom loss function that calculates the distance of a "mixed" image from the prototypes of its constituent classes, weighted by the mixing ratio. This encourages the model to cluster similar images tightly around their class prototypes, creating a more structured feature space where OOD samples are easier to identify as outliers.

Experimental Setup

We evaluated our approach using three primary datasets:

  • Molemap (In-house): A massive, realistic tele-dermatology dataset containing 208,287 images across 65 conditions, exhibiting extreme long-tailed and fine-grained characteristics.
  • ISIC 2019: A standard public benchmark re-partitioned to simulate a long-tailed clinical scenario.
  • CIFAR-10: Used as a coarse-grained OOD baseline to demonstrate the performance gap between identifying non-medical images versus unknown medical conditions.

Key Results

Our proposed M-T Mixup + Prototype approach significantly outperformed state-of-the-art methods such as ODIN, OLTR, and ARPL.

  • OOD Detection Performance: On the challenging Molemap dataset, OOD detection (measured by AUROC) improved from a baseline of 65.67% to 82.71%.
  • Accuracy Preservation: Unlike many OOD methods that sacrifice performance on known classes, our approach maintained or slightly improved the F1-score for In-Distribution (ID) categories.
  • Robustness: The model showed enhanced capability in identifying "unusual" OOD samples, such as images obscured by hair or motion blur, which are common in real-world clinical photography.

Discussion and Conclusion

The results highlight that domain-specific OOD detection is significantly harder than standard benchmarks. While existing methods excel at spotting a truck in a medical dataset, they struggle to distinguish between two similar-looking skin conditions. By focusing on the "Middle" and "Tail" classes and utilizing prototype-based clustering, we provide the model with better "boundary awareness."

Despite these gains, challenges remain for clinical deployment. The current partitioning of classes is manual, and future work should focus on automating this process. Additionally, further validation is needed to ensure robustness across different camera hardwares and lighting conditions. Nevertheless, this approach represents a significant step toward more reliable and safe automated dermatology systems.

Summary updated

Abstract

Recent years have witnessed a rapid development of automated methods for skin lesion diagnosis and classification. Due to an increasing deployment of such systems in clinics, it has become important to develop a more robust system towards various Out-of-Distribution(OOD) samples (unknown skin lesions and conditions). However, the current deep learning models trained for skin lesion classification tend to classify these OOD samples incorrectly into one of their learned skin lesion categories. To address this issue, we propose a simple yet strategic approach that improves the OOD detection performance while maintaining the multi-class classification accuracy for the known categories of skin lesion. To specify, this approach is built upon a realistic scenario of a long-tailed and fine-grained OOD detection task for skin lesion images. Through this approach, 1) First, we target the mixup amongst middle and tail classes to address the long-tail problem. 2) Later, we combine the above mixup strategy with prototype learning to address the fine-grained nature of the dataset. The unique contribution of this paper is two-fold, justified by extensive experiments. First, we present a realistic problem setting of OOD task for skin lesion. Second, we propose an approach to target the long-tailed and fine-grained aspects of the problem setting simultaneously to increase the OOD performance.
BibTeX
@inproceedings{mehta2022out,
  author = {Deval Mehta and Yaniv Gal and Adrian Bowling and C. Paul Bonnington and Zongyuan Ge},
  title = {{Out-of-Distribution Detection for Long-tailed and Fine-grained Skin Lesion Images}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  url = {https://openreview.net/forum?id=ucpELr7j5s4},
}
Figure