Efficient Out-of-Distribution Detection of Melanoma with Wavelet-based Normalizing Flows

M. M. Amaan Valiuddin, Christiaan G. A. Viviers, Ruud J. G. van Sloun, Peter H. N. de With, Fons van der Sommen

Abstract

Melanoma is a serious form of skin cancer with high mortality rate at later stages. Fortunately, when detected early, the prognosis of melanoma is promising and malignant melanoma incidence rates are relatively low. As a result, datasets are heavily imbalanced which complicates training current state-of-the-art supervised classification AI models. We propose to use generative models to learn the benign data distribution and detect Out-of-Distribution (OOD) malignant images through density estimation. Normalizing Flows (NFs) are ideal candidates for OOD detection due to their ability to compute exact likelihoods. Nevertheless, their inductive biases towards apparent graphical features rather than semantic context hamper accurate OOD detection. In this work, we aim at using these biases with domain-level knowledge of melanoma, to improve likelihood-based OOD detection of malignant images. Our encouraging results demonstrate potential for OOD detection of melanoma using NFs. We achieve a 9% increase in Area Under Curve of the Receiver Operating Characteristics by using wavelet-based NFs. This model requires significantly less parameters for inference making it more applicable on edge devices. The proposed methodology can aid medical experts with diagnosis of skin-cancer patients and continuously increase survival rates. Furthermore, this research paves the way for other areas in oncology with similar data imbalance issues.

Introduction

Melanoma represents one of the most lethal forms of skin cancer, yet it remains highly treatable if identified in its early stages. A significant hurdle in developing automated diagnostic tools is the inherent imbalance in clinical datasets; benign cases vastly outnumber malignant ones. Traditional supervised deep learning models often struggle to generalize under these conditions, as they require large, balanced datasets to distinguish between classes effectively.

This research proposes an alternative paradigm: treating melanoma detection as an Out-of-Distribution (OOD) detection problem. By training a model exclusively on the distribution of benign skin samples, malignant tumors can be identified as anomalies that deviate from the learned "normal" distribution.

Normalizing Flows and the Texture Bias

Normalizing Flows (NFs) are a class of generative models capable of computing exact likelihoods, making them theoretically ideal for OOD detection. However, NFs often exhibit a counterintuitive inductive bias: they tend to prioritize low-level graphical features, such as local pixel correlations and textures, over high-level semantic content. In many computer vision tasks, this leads the model to assign higher likelihoods to OOD data than to the training data.

In the context of dermatology, however, this "bug" can be leveraged as a feature. Malignant melanoma is characterized by complex textures and irregular borders—features that NFs are naturally sensitive to. By refining how the model processes these textures, the authors aim to turn the NF's inherent biases into a diagnostic advantage.

Methodology: Wavelet-based Decomposition

The core of the proposed approach is the integration of Wavelet Flow. Rather than processing raw RGB pixels, the architecture utilizes the Haar wavelet transform to decompose images into multiple scales, separating low-frequency structural information from high-frequency details and textures.

Key methodological choices include:

  1. Grayscale Conversion: Images were converted to grayscale to prevent the model from over-relying on color-pixel correlations, which often lead to spurious likelihood assignments in OOD tasks.
  2. Masking Strategies: The researchers implemented specific coupling layer masks (e.g., Radial and Cycle masking) to prevent "co-adaptation," where the model simply memorizes local pixel patterns rather than learning the broader distribution.
  3. Scale-Specific Likelihoods: The final OOD score is calculated by averaging likelihoods across specific wavelet decomposition scales. The study found that the 3rd through 6th scales—representing mid-range structural features—were the most effective for detection.

Experimental Results

The model was trained and evaluated using the ISIC (International Skin Imaging Collaboration) dataset. The training set consisted solely of benign images, while the evaluation set included both benign (In-Distribution) and malignant (OOD) samples.

The Wavelet-based NF demonstrated significant improvements over the baseline Glow model:

  • Detection Accuracy: The approach achieved a 9% increase in the Area Under the Curve (AUC) of the Receiver Operating Characteristics, reaching a peak AUC of 0.78.
  • Model Efficiency: The Wavelet Flow model is exceptionally lean. While a standard RGB Glow baseline requires approximately 159 million parameters, the proposed model achieves superior results with only 1.25 million parameters. This ~127x reduction in size makes the model highly suitable for deployment on edge devices and mobile medical hardware.

Discussion and Future Directions

While the results are promising, the model faces a specific challenge known as the "Hair Problem." Because NFs assign lower likelihoods to complex textures, benign moles obscured by hair are sometimes incorrectly flagged as malignant (OOD).

Future research will focus on incorporating a "complexity correction term" to account for non-cancerous textures like hair. Additionally, the success of this wavelet-based OOD approach suggests potential applications in other oncological fields where data is heavily imbalanced. By focusing on the efficiency and specific inductive biases of Normalizing Flows, this methodology paves the way for accessible, real-time skin cancer screening.

Summary updated

Abstract

Melanoma is a serious form of skin cancer with high mortality rate at later stages. Fortunately, when detected early, the prognosis of melanoma is promising and malignant melanoma incidence rates are relatively low. As a result, datasets are heavily imbalanced which complicates training current state-of-the-art supervised classification AI models. We propose to use generative models to learn the benign data distribution and detect Out-of-Distribution (OOD) malignant images through density estimation. Normalizing Flows (NFs) are ideal candidates for OOD detection due to their ability to compute exact likelihoods. Nevertheless, their inductive biases towards apparent graphical features rather than semantic context hamper accurate OOD detection. In this work, we aim at using these biases with domain-level knowledge of melanoma, to improve likelihood-based OOD detection of malignant images. Our encouraging results demonstrate potential for OOD detection of melanoma using NFs. We achieve a 9% increase in Area Under Curve of the Receiver Operating Characteristics by using wavelet-based NFs. This model requires significantly less parameters for inference making it more applicable on edge devices. The proposed methodology can aid medical experts with diagnosis of skin-cancer patients and continuously increase survival rates. Furthermore, this research paves the way for other areas in oncology with similar data imbalance issues.
BibTeX
@inproceedings{valiuddin2022efficient,
  author = {M. M. Amaan Valiuddin and Christiaan G. A. Viviers and Ruud J. G. van Sloun and Peter H. N. de With and Fons van der Sommen},
  title = {{Efficient Out-of-Distribution Detection of Melanoma with Wavelet-based Normalizing Flows}},
  booktitle = {CaPTion @ MICCAI},
  year = {2022},
  url = {https://openreview.net/forum?id=v_Y7Y_jRNZ},
}
Figure