TL;DR
This paper introduces MICA, a novel framework for explainable skin lesion diagnosis that aligns medical images with clinical concepts at multiple levels (image, token, and concept). By leveraging multi-level alignment and a medical LLM for concept understanding, MICA achieves superior diagnostic performance and interpretability compared to existing methods, offering both visual and textual explanations.
What Problem Does This Paper Address?
Deep learning models have shown great promise in medical image analysis, but their "black-box" nature poses a significant challenge for trust and adoption in healthcare. Explainable AI (XAI) aims to address this, with concept-based methods being particularly attractive for their ability to link model decisions to human-understandable concepts. However, existing concept-based methods often rely on single-level concept annotations (e.g., global image level), failing to capture the nuanced semantic relationships between image sub-regions and concepts. This can lead to underutilization of valuable medical information, hindering both model performance and interpretability. The paper aims to bridge this gap by proposing a framework that aligns medical images and clinical concepts at multiple semantic levels.
Methods
MICA employs a two-stage approach:
- Multi-Level Image-Concept Alignment: This stage focuses on learning robust correspondences between medical images and clinical concepts.
- Image-Level Alignment (ILA): Maximizes the similarity between global image representations and concept representations, treating them as pairs.
- Token-Level Alignment (TLA): Utilizes an attention mechanism to cultivate fine-grained alignments between image sub-regions (tokens) and concept word tokens. This is crucial for medical imaging where specific regions often hold diagnostic significance.
- Concept-Level Alignment (CLA): Leverages Concept Activation Vectors (CAVs) to map aggregated image representations into a concept subspace, further refining the alignment with ground truth concepts.
- Medical LLM Integration: A key innovation is the use of a medical Large Language Model (LLM) as a concept encoder. This allows the model to understand the latent conceptual semantics of clinical concepts, going beyond simple word embeddings.
- Explainable Disease Diagnosis: This stage mimics how medical experts diagnose.
- Concept Prediction: The trained image encoder is used to predict clinical concepts present in an image.
- Linear Probe Diagnosis: A simple linear layer then predicts the final disease diagnosis based on these detected concepts. This makes the final decision process inherently interpretable, as it's directly tied to the identified concepts.
- Explanations: The framework generates both visual (localization of concepts) and textual explanations.
Datasets
The paper evaluates MICA on three skin image datasets:
- Derm7pt: A dermatology dataset with 1011 images and clinical concepts for melanoma diagnosis.
- PH²: A dataset of 200 dermoscopic images of melanocytic lesions.
- SkinCon: A skin disease dataset with 3230 images annotated by experts for fine-grained model debugging and analysis.
Key Insights and Ideas
- Multi-level alignment is crucial: Capturing semantic relationships at image, token, and concept levels significantly improves both diagnostic accuracy and interpretability, moving beyond single-level approaches.
- LLMs unlock deeper concept understanding: Integrating a medical LLM for concept encoding allows the model to grasp richer semantic meanings of clinical concepts, enhancing the alignment process.
- Mimicking expert diagnosis: The two-stage approach (concept prediction followed by diagnosis) closely mirrors how human experts work, leading to more trustworthy and understandable AI.
- Ante-hoc interpretability is preferred: The paper advocates for building interpretability into the model from the start (ante-hoc) rather than trying to explain a black-box model after training (post-hoc), which can be unreliable.
Key Results
- Superior Disease Diagnosis: MICA achieves state-of-the-art performance across three datasets (Derm7pt, PH², SkinCon) in terms of AUC, Accuracy, and F1 Score, often outperforming existing concept-based methods. For instance, on Derm7pt, MICA achieves an AUC of 78.6%, compared to 76.2% for Sarkar et al. (2022).
- High Concept Detection Accuracy: The method also demonstrates strong performance in detecting clinical concepts, as shown in Table 2 and Table 3.
- Label Efficiency: MICA shows robust performance even with reduced amounts of training data for the diagnosis stage, indicating efficient learning from concept annotations.
- Faithful Explanations: Test-time intervention experiments demonstrate that the model's predictions are indeed faithful to the concepts it identifies, and changing concept scores alters the diagnosis accordingly.
Limitations and Future Work
- LLM Parameter Freezing: For efficiency, the LLM parameters are fixed during training. The paper notes this as a potential limitation and suggests future work could explore fine-tuning the LLM.
- LLM Accuracy: The paper acknowledges that not all knowledge derived from the LLM is guaranteed to be accurate, which could impact the model.
- Future Work: The authors suggest further research into improving the LLM integration and exploring more advanced concept-based explanation techniques. They also highlight the potential for MICA to be applied to other medical imaging domains.
Summary updated