Abstract
MICA (Multi-Level Image-Concept Alignment) is a novel framework designed to bridge the gap between high-performance deep learning and clinical interpretability in skin lesion diagnosis. By aligning images and medical concepts across three granularities—global, regional, and conceptual—MICA provides accurate diagnoses alongside clear visual and textual justifications. It leverages medical-specific Large Language Models (LLMs) and Concept Activation Vectors to ensure that learned features correspond to actual clinical symptoms, enabling human-in-the-loop interventions.
Introduction
Deep learning models have shown significant promise in medical imaging but often function as "black boxes," providing diagnoses without clinical reasoning. In dermatology, where identifying specific features like "pigment networks" or "irregular streaks" is crucial for trust and verification, this lack of transparency is a significant hurdle. Existing Concept Bottleneck Models (CBMs) attempt to address this by predicting human-understandable concepts before making a final diagnosis. However, these models often suffer from a performance gap compared to black-box models, largely because they fail to account for the localized nature of dermatological symptoms.
Methods
MICA introduces a two-stage approach to align visual features with clinical knowledge at multiple levels of granularity:
- Multi-Level Alignment:
- Image Level: Matches global visual features to overall concept descriptions.
- Token Level: Uses a cross-attention mechanism to link specific image patches (sub-regions) to specific words (tokens) in the concept definitions, helping the model "locate" symptoms.
- Concept Level: Employs Concept Activation Vectors (CAVs) to project image features into a dedicated clinical subspace, ensuring learned features represent actual symptoms.
- Medical LLM Integration: MICA is the first framework to use a medical-specific LLM (ClinicalBERT) as a concept encoder, leveraging pre-trained semantic relationships between complex dermatological terms.
- Explainable Bottleneck: A linear probe predicts the diagnosis based solely on detected concepts. This architecture allows for human-in-the-loop intervention, where doctors can manually correct predicted concepts to update the diagnosis.
Datasets
The framework was evaluated on three prominent datasets:
- Derm7pt: 1,011 dermoscopic images with 7-point checklist annotations.
- PH²: 200 high-quality images focused on melanocytic lesions.
- SkinCon: 3,230 clinical images annotated with 22 diverse concepts, representing real-world scenarios.
Key Results
MICA demonstrated significant improvements in both accuracy and interpretability:
- Superior Accuracy: On the Derm7pt dataset, MICA achieved an AUC of 84.11%, significantly outperforming other state-of-the-art explainable models like CBE (76.60%) and PCBM (72.96%).
- Better Concept Detection: MICA outperformed existing methods in identifying clinical symptoms across all three datasets.
- Intervention and Faithfulness: Experiments proved that model decisions are logically tied to concepts. Manually adjusting concept scores (e.g., removing "irregular streaks") resulted in predictable, medically sound changes in diagnosis.
- Visual & Textual Explanations: The model generates heatmaps for symptom localization and provides percentage-based textual justifications for its findings.
Discussion and Limitations
While MICA marks a significant step forward, several areas for improvement remain. The medical LLM was kept frozen to save computational resources; fine-tuning it on dermatological literature could further enhance semantic understanding. Additionally, like all CBMs, MICA depends on high-quality expert annotations, which are expensive to collect. Future work will explore extending this multi-level alignment to other complex medical fields such as histopathology or radiology.
Summary updated