Introduction
The Segment Anything Model (SAM) has revolutionized computer vision with its zero-shot capabilities. However, its application in clinical medical imaging is hindered by its reliance on manual prompts, such as points or bounding boxes. In high-throughput medical environments, radiologists lack the time to manually prompt every image. While previous "self-prompting" attempts have sought to automate this process, they primarily focus on positional information, which often fails to generalize across the diverse modalities and blurry boundaries characteristic of medical scans.
This paper introduces Hierarchical Self-Prompting SAM (HSP-SAM), a framework that replaces manual intervention with learned "abstract task-guided prompts." By moving beyond simple positional coordinates, HSP-SAM achieves a more robust, semantic understanding of medical segmentation tasks, enabling fully automated performance that generalizes across unseen datasets.
Methodology
HSP-SAM incorporates three primary architectural innovations to transform SAM into a prompt-free medical tool:
- Q&A-based Self-Prompt Module: Instead of predicting spatial coordinates, the model learns "Question" (Q) and "Answer" (A) prompt pairs. Q-prompts are injected into the image encoder to identify task-relevant features, while A-prompts guide the mask decoder in delineating targets. These pairs are connected via a bottleneck structure that captures task-specific semantic information.
- Hierarchical Feature Fusion: To address the challenge of blurry boundaries in medical images, the authors implemented a U-shaped structure. This design fuses high-dimensional semantic features from the encoder with low-dimensional structural details across multiple layers, ensuring precise boundary localization.
- Parameter-Efficient Fine-tuning: To adapt SAM’s massive 1-billion-parameter encoder without prohibitive costs, the framework utilizes Low-Rank Adaptation (LoRA). This allows the model to be tuned specifically for medical tasks while keeping the majority of the original weights frozen.
Experimental Evaluation
The researchers evaluated HSP-SAM across several challenging benchmarks:
- Polyp Segmentation: Tested on Kvasir-SEG, CVC-ClinicDB, CVC-300, CVC-ColonDB, and ETIS-LaribDB.
- Skin Lesion Segmentation: Evaluated using ISIC 2017 and ISIC 2018 datasets.
- Multi-Modality Generalization: The model was tested on five source/target pairs (e.g., training on DRIVE and testing on the unseen STARE dataset) to assess zero-shot performance on entirely new data types.
Results and Discussion
HSP-SAM consistently outperformed both the original SAM and specialized medical segmentation models. On the CVC-ClinicDB dataset, it achieved a state-of-the-art 95.86% Dice score, surpassing models like PolypPVT.
Key insights from the study include:
- Abstract over Positional: Teaching the model the concept of a task through abstract prompts proved far more effective for generalization than teaching it where an object is located.
- Zero-Shot Generalization: In tests on unseen data, HSP-SAM improved Dice scores by up to 14.04% over previous state-of-the-art methods, demonstrating remarkable robustness.
- Attention Specialization: Visualizations revealed that Q-prompts naturally learn to focus on foreground objects while A-prompts focus on the background, a division of labor that emerges automatically during training.
- Boundary Precision: The model achieved significantly lower Hausdorff Distance (HD) scores, indicating that the predicted boundaries are much closer to the ground truth than those of competing models.
Conclusion
HSP-SAM represents a significant step toward fully automated medical image analysis. By replacing manual prompts with a hierarchical, self-prompting mechanism, the framework achieves high accuracy and superior generalization. Future work may involve extending this 2D framework to 3D volumetric data (CT/MRI) and exploring the integration of linguistic representations to allow clinicians to define new tasks through natural language.
Summary updated