Introduction
Deep learning models in medical diagnostics often operate as "black boxes," achieving high accuracy but lacking transparency in their decision-making processes. Concept Bottleneck Models (CBMs) aim to address this by predicting human-understandable concepts before a final diagnosis. However, existing CBMs frequently suffer from "layer tunnel vision"—focusing only on the final layer—and "knowledge forgetting" during fine-tuning.
CoPA (Concept Prompting and Aggregating) is a novel framework that utilizes hierarchical prompting and a frozen backbone to map complex image data to clinical concepts across multiple neural network layers. By "interrogating" the vision model at every stage, CoPA ensures that the model's reasoning aligns with clinical logic while maintaining state-of-the-art performance.
Methodology: The CoPA Architecture
CoPA's architecture is built around three primary innovations designed to capture multiscale features and maintain clinical relevance while leveraging a frozen pre-trained backbone (BiomedCLIP).
Concept-aware Embedding Generator (CEG)
Rather than relying on generic layer outputs, the CEG employs learnable "concept anchors." These anchors act as queries to extract specific information related to medical concepts (e.g., "pigment dots" or "irregular networks") from each layer of the transformer backbone.
Concept Prompt Tuning (CPT)
To maintain focus across the network, embeddings from one layer's CEG are passed to the subsequent layer as "prompts." This hierarchical guidance encourages the frozen backbone to progressively amplify visual cues relevant to the final diagnosis, effectively "steering" the model's attention.
Gated Aggregation and Frozen Backbone
CoPA keeps the main vision encoder weights frozen to prevent the loss of general visual knowledge. A final gating network learns to weight the importance of different concepts for specific diseases, mimicking the diagnostic process of a medical specialist who weighs various clinical signs.
Experimental Evaluation
The framework was evaluated on three dermoscopy datasets: PH² (200 images, 5 features), Derm7pt (1,011 cases, 7-point checklist), and SkinCon (3,690 images, 22 features).
Performance and Results
CoPA consistently outperformed state-of-the-art explainable models, including PCBM and MICA, across all metrics.
Key findings include:
- Diagnostic Accuracy: On the Derm7pt dataset, CoPA achieved a 2.1% improvement in accuracy over previous benchmarks.
- Concept Mapping: Concept prediction accuracy on the PH² dataset improved by 2.6%.
- Faithfulness: In intervention experiments—where human experts manually corrected predicted concepts—the model's final diagnostic accuracy increased by up to 1.1%, confirming that the diagnosis is genuinely grounded in the intermediate concepts.
Discussion and Insights
The success of CoPA highlights that medical concepts are inherently multiscale; critical diagnostic signs often reside in low-level textures that are lost in deeper layers. By using prompting as a guide, the model effectively focuses on clinically significant regions.
Despite its performance, CoPA's reliance on expert-annotated concept datasets remains a bottleneck, as such data is expensive to produce. Additionally, the hierarchical processing of every layer introduces higher computational overhead compared to single-layer models. Future research will explore the generalization of this hierarchical prompting approach to 3D medical imaging modalities such as MRI and CT scans.
Summary updated