Out-of-Distribution Detection for Long-Tailed and Fine-Grained Skin Lesion Images

Deval Mehta, Yaniv Gal, Adrian Bowling, C. Paul Bonnington, Zongyuan Ge

TL;DR

This paper introduces a strategic approach to Out-of-Distribution (OOD) detection for skin lesion classification by combining targeted Mixup data augmentation with Prototype Learning. By focusing these techniques on the "middle" and "tail" classes of a long-tailed dataset, the authors significantly improve the model's ability to reject unknown skin conditions without sacrificing accuracy on known ones.

What Problem Does This Paper Address?

Deep learning models are increasingly used in dermatology, but they suffer from a "confidence problem." When shown a skin condition they haven't seen before (an OOD sample), these models often confidently misclassify it as a known disease rather than flagging it as unknown. This is dangerous in a clinical setting.

The authors argue that current OOD research is flawed because it relies on "toy" datasets like CIFAR-10, where classes are balanced and visually distinct (e.g., a car vs. a dog). Real-world clinical data is much harder:

  1. Long-tailed: A few common conditions have thousands of images, while many rare diseases have very few.
  2. Fine-grained: Different skin lesions look incredibly similar, making the "boundaries" between classes blurry.

Standard OOD methods fail here because the model's feature space is dominated by the common "head" classes, leaving the rare "tail" classes—and the OOD boundaries—poorly defined.

Methods

The authors propose a two-pronged strategy to sharpen these decision boundaries:

  1. Targeted Mixup (M-T Intersubset): Mixup is a technique where two images and their labels are blended together to teach the model about the space between categories. The authors found that standard Mixup is dominated by common classes. Instead, they partitioned their data into Head (common), Middle, and Tail (rare) subsets. Their key innovation was focusing Mixup specifically between Middle and Tail classes (MX5 strategy). This forces the model to learn better features for rare conditions, which are often the ones most easily confused with OOD samples.
  2. Prototype Learning Integration: To handle the fine-grained nature of the data, they integrated Mixup with Prototype Learning. In this setup, the model learns a "prototype" (a representative point in feature space) for every class. They developed a custom loss function that calculates the distance of a "mixed" image from the prototypes of its constituent classes, weighted by the mixing ratio. This encourages the model to cluster similar images tightly around their class prototypes.

Datasets

The study utilizes three primary datasets:

  • Molemap (In-house): A massive, realistic dataset of 208,287 tele-dermatology images across 65 conditions. It is highly long-tailed and fine-grained.
  • ISIC 2019: A standard public benchmark for skin lesions, which the authors re-partitioned to simulate a long-tailed clinical scenario.
  • CIFAR-10: Used as a "coarse-grained" OOD baseline to show how much easier it is to detect "non-medical" images compared to "unknown medical" images.

Key Insights and Ideas

  • The "Head" Class Bias: Standard data augmentation often helps the classes that already have plenty of data. By restricting Mixup to the "Middle" and "Tail" subsets, the authors effectively "leveled the playing field" for the model's feature extractor.
  • Domain-Specific OOD is Hard: The paper demonstrates that while existing methods (like ODIN or MSP) are great at spotting a picture of a truck in a skin lesion dataset (OOD-Cifar), they struggle significantly when the OOD sample is just a different, unknown skin condition.
  • Boundary Awareness: By combining prototypes with Mixup, the model doesn't just learn what a disease is; it learns exactly where that disease ends in the feature space. This "boundary awareness" is what drives the improved OOD detection.

Key Results

The proposed M-T Mixup + Prototype approach outperformed several state-of-the-art methods (including ODIN, OLTR, and ARPL):

  • OOD Detection: On the challenging in-house dataset, the OOD detection performance (measured by AUROC) jumped from a baseline of 65.67% to 82.71%.
  • Accuracy Preservation: Unlike many OOD methods that degrade performance on known classes, this approach maintained or slightly improved the F1-score for the "In-Distribution" (ID) categories.
  • Robustness to "Unusual" Noise: The model also showed improved performance in identifying "unusual" OOD samples, such as blurred images or those obscured by hair, which are common in real-world clinical photography.

Limitations and Future Work

While the paper presents a significant step forward, some limitations remain:

  • Hyperparameter Sensitivity: The partitioning of Head, Middle, and Tail classes is currently manual/heuristic. Automating this split based on data distribution could make the method more generalizable.
  • Clinical Validation: While the "Molemap" dataset is realistic, the authors note that the current techniques are "still far away from clinical deployment." Future work likely needs to address how these models handle hardware variations (different cameras/lighting) across different clinics.
  • Computational Overhead: Prototype learning requires maintaining and updating class centers, which adds complexity to the training loop compared to standard cross-entropy.

Summary updated

Abstract

Recent years have witnessed a rapid development of automated methods for skin lesion diagnosis and classification. Due to an increasing deployment of such systems in clinics, it has become important to develop a more robust system towards various Out-of-Distribution (OOD) samples (unknown skin lesions and conditions). However, the current deep learning models trained for skin lesion classification tend to classify these OOD samples incorrectly into one of their learned skin lesion categories. To address this issue, we propose a simple yet strategic approach that improves the OOD detection performance while maintaining the multi-class classification accuracy for the known categories of skin lesion. To specify, this approach is built upon a realistic scenario of a long-tailed and fine-grained OOD detection task for skin lesion images. Through this approach, 1) First, we target the mixup amongst middle and tail classes to address the long-tail problem. 2) Later, we combine the above mixup strategy with prototype learning to address the fine-grained nature of the dataset. The unique contribution of this paper is two-fold, justified by extensive experiments. First, we present a realistic problem setting of OOD task for skin lesion. Second, we propose an approach to target the long-tailed and fine-grained aspects of the problem setting simultaneously to increase the OOD performance.
BibTeX
@inproceedings{mehta2022out,
  author = {Deval Mehta and Yaniv Gal and Adrian Bowling and C. Paul Bonnington and Zongyuan Ge},
  title = {{Out-of-Distribution Detection for Long-Tailed and Fine-Grained Skin Lesion Images}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  url = {https://openreview.net/forum?id=ucpELr7j5s4},
}

Loading related content...

Figure