Concept-Attention Whitening for Interpretable Skin Lesion Diagnosis

Junlin Hou, Jilan Xu, Hao Chen

Abstract

The black-box nature of deep learning models has raised concerns about their interpretability for successful deployment in real-world clinical applications. To address the concerns, eXplainable Artificial Intelligence (XAI) aims to provide clear and understandable explanations of the decision-making process. In the medical domain, concepts such as attributes of lesions or abnormalities serve as key evidence for deriving diagnostic results. Existing concept-based models mainly depend on concepts that appear independently and require fine-grained concept annotations such as bounding boxes. However, a medical image usually contains multiple concepts, and the fine-grained concept annotations are difficult to acquire. In this paper, we aim to interpret representations in deep neural networks by aligning the axes of the latent space with known concepts of interest. We propose a novel Concept-Attention Whitening (CAW) framework for interpretable skin lesion diagnosis. CAW is comprised of a disease diagnosis branch and a concept alignment branch. In the former branch, we train a convolutional neural network (CNN) with an inserted CAW layer to perform skin lesion diagnosis. The CAW layer decorrelates features and aligns image features to conceptual meanings via an orthogonal matrix. In the latter branch, the orthogonal matrix is calculated under the guidance of the concept attention mask. We particularly introduce a weakly-supervised concept mask generator that only leverages coarse concept labels for filtering local regions that are relevant to certain concepts, improving the optimization of the orthogonal matrix. Extensive experiments on two public skin lesion diagnosis datasets demonstrated that CAW not only enhanced interpretability but also maintained a state-of-the-art diagnostic performance.

Introduction

Deep learning models have achieved remarkable success in medical imaging, yet their "black-box" nature remains a significant barrier to clinical adoption. In dermatology, clinicians rely on specific visual attributes, such as the "7-point checklist," to diagnose skin lesions. While concept-based eXplainable AI (XAI) attempts to bridge this gap by linking model activations to medical attributes, existing methods struggle with images containing multiple overlapping concepts and typically require labor-intensive, fine-grained annotations like bounding boxes.

This paper introduces Concept-Attention Whitening (CAW), a framework designed to provide interpretable skin lesion diagnosis by aligning the latent space of neural networks with clinical concepts. CAW addresses the limitations of previous models by handling multi-concept interference and utilizing only coarse, image-level labels for training, ensuring that interpretability does not come at the cost of diagnostic accuracy.

Methodology

The CAW framework employs a unique dual-branch optimization strategy to achieve both diagnostic accuracy and interpretability.

The CAW framework architecture, illustrating the dual-branch optimization strategy and the integration of the Concept-Attention Whitening layer.

The CAW Layer

The core innovation is the Concept-Attention Whitening layer, which replaces standard Batch Normalization in a CNN. This layer performs two key functions:

  1. Whitening: It decorrelates internal features, ensuring that each dimension in the latent space is independent.
  2. Orthogonal Transformation: It rotates the feature space to align specific axes with known clinical concepts via an orthogonal matrix $Q$.

Weakly-Supervised Concept Mask Generator

To handle images with multiple overlapping features, CAW introduces a weakly-supervised concept mask generator. This component uses a pre-trained concept classifier to produce activation maps that highlight regions relevant to specific concepts. These masks act as filters, allowing the model to focus on local evidence (e.g., specific areas of "blue whitish veil") during the optimization of the orthogonal matrix, effectively denoising the alignment process.

Experimental Results

The framework was validated on two major public datasets: Derm7pt (827 images with 12 concepts) and SkinCon (3,230 images with 48 concepts).

Diagnostic and Interpretability Performance

CAW maintains high diagnostic performance, achieving an AUC of 88.60% on Derm7pt, nearly matching unconstrained "black-box" models (89.48%). Crucially, it excels in concept detection, outperforming methods like TCAV and standard Concept Whitening. CAW reached an average AUC of 77.4% on Derm7pt and 78.1% on SkinCon, demonstrating its superior ability to isolate clinical features within its latent space.

Visual and Textual Explanations

A key advantage of CAW is its ability to provide human-readable explanations. By calculating "Concept Importance Scores," the model can quantify the contribution of specific clinical features to a diagnosis. This allows the system to provide both visual masks and textual justifications for its clinical decisions.

Visual and textual explanation generated by CAW for a melanoma diagnosis, showing localized concept masks and their relative importance.

Discussion and Conclusion

Concept-Attention Whitening demonstrates that interpretability can be achieved without sacrificing diagnostic power. By using attention masks as a denoising tool, the model effectively disentangles complex medical features using only coarse annotations. Future work will explore "softening" the orthogonality constraints to better represent naturally correlated medical concepts and adapting the framework for Transformer-based architectures.

Summary updated

Abstract

The black-box nature of deep learning models has raised concerns about their interpretability for successful deployment in real-world clinical applications. To address the concerns, eXplainable Artificial Intelligence (XAI) aims to provide clear and understandable explanations of the decision-making process. In the medical domain, concepts such as attributes of lesions or abnormalities serve as key evidence for deriving diagnostic results. Existing concept-based models mainly depend on concepts that appear independently and require fine-grained concept annotations such as bounding boxes. However, a medical image usually contains multiple concepts, and the fine-grained concept annotations are difficult to acquire. In this paper, we aim to interpret representations in deep neural networks by aligning the axes of the latent space with known concepts of interest. We propose a novel Concept-Attention Whitening (CAW) framework for interpretable skin lesion diagnosis. CAW is comprised of a disease diagnosis branch and a concept alignment branch. In the former branch, we train a convolutional neural network (CNN) with an inserted CAW layer to perform skin lesion diagnosis. The CAW layer decorrelates features and aligns image features to conceptual meanings via an orthogonal matrix. In the latter branch, the orthogonal matrix is calculated under the guidance of the concept attention mask. We particularly introduce a weakly-supervised concept mask generator that only leverages coarse concept labels for filtering local regions that are relevant to certain concepts, improving the optimization of the orthogonal matrix. Extensive experiments on two public skin lesion diagnosis datasets demonstrated that CAW not only enhanced interpretability but also maintained a state-of-the-art diagnostic performance.
BibTeX
@inproceedings{hou2024concept,
  author = {Junlin Hou and Jilan Xu and Hao Chen},
  title = {{Concept-Attention Whitening for Interpretable Skin Lesion Diagnosis}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  url = {https://openreview.net/forum?id=U9HAF2Aybm},
}
Figure