Hierarchical Self-Prompting SAM: A Prompt-Free Medical Image Segmentation Framework

Mengmeng Zhang, Xingyuan Dai, Yicheng Sun, Jing Wang, Yueyang Yao, Xiaoyan Gong, Fu'ze Cong, Fei-Yue Wang, Yisheng LvarXiv 2025 · DOI · arXiv

Introduction

The Segment Anything Model (SAM) has revolutionized computer vision with its zero-shot capabilities. However, its application in clinical medical imaging is hindered by its reliance on manual prompts, such as points or bounding boxes. In high-throughput medical environments, radiologists lack the time to manually prompt every image. While previous "self-prompting" attempts have sought to automate this process, they primarily focus on positional information, which often fails to generalize across the diverse modalities and blurry boundaries characteristic of medical scans.

Comparison of prompt-based methods: (a) Manual prompts, (b) Previous self-prompt methods focusing on positional info, and (c) The proposed HSP-SAM using abstract task-guided prompts.

This paper introduces Hierarchical Self-Prompting SAM (HSP-SAM), a framework that replaces manual intervention with learned "abstract task-guided prompts." By moving beyond simple positional coordinates, HSP-SAM achieves a more robust, semantic understanding of medical segmentation tasks, enabling fully automated performance that generalizes across unseen datasets.

Methodology

HSP-SAM incorporates three primary architectural innovations to transform SAM into a prompt-free medical tool:

  1. Q&A-based Self-Prompt Module: Instead of predicting spatial coordinates, the model learns "Question" (Q) and "Answer" (A) prompt pairs. Q-prompts are injected into the image encoder to identify task-relevant features, while A-prompts guide the mask decoder in delineating targets. These pairs are connected via a bottleneck structure that captures task-specific semantic information.
  2. Hierarchical Feature Fusion: To address the challenge of blurry boundaries in medical images, the authors implemented a U-shaped structure. This design fuses high-dimensional semantic features from the encoder with low-dimensional structural details across multiple layers, ensuring precise boundary localization.
  3. Parameter-Efficient Fine-tuning: To adapt SAM’s massive 1-billion-parameter encoder without prohibitive costs, the framework utilizes Low-Rank Adaptation (LoRA). This allows the model to be tuned specifically for medical tasks while keeping the majority of the original weights frozen.
The detailed architecture of HSP-SAM, highlighting the hierarchical structure and the Q&A-based self-prompt module.

Experimental Evaluation

The researchers evaluated HSP-SAM across several challenging benchmarks:

  • Polyp Segmentation: Tested on Kvasir-SEG, CVC-ClinicDB, CVC-300, CVC-ColonDB, and ETIS-LaribDB.
  • Skin Lesion Segmentation: Evaluated using ISIC 2017 and ISIC 2018 datasets.
  • Multi-Modality Generalization: The model was tested on five source/target pairs (e.g., training on DRIVE and testing on the unseen STARE dataset) to assess zero-shot performance on entirely new data types.

Results and Discussion

HSP-SAM consistently outperformed both the original SAM and specialized medical segmentation models. On the CVC-ClinicDB dataset, it achieved a state-of-the-art 95.86% Dice score, surpassing models like PolypPVT.

Key insights from the study include:

  • Abstract over Positional: Teaching the model the concept of a task through abstract prompts proved far more effective for generalization than teaching it where an object is located.
  • Zero-Shot Generalization: In tests on unseen data, HSP-SAM improved Dice scores by up to 14.04% over previous state-of-the-art methods, demonstrating remarkable robustness.
  • Attention Specialization: Visualizations revealed that Q-prompts naturally learn to focus on foreground objects while A-prompts focus on the background, a division of labor that emerges automatically during training.
  • Boundary Precision: The model achieved significantly lower Hausdorff Distance (HD) scores, indicating that the predicted boundaries are much closer to the ground truth than those of competing models.

Conclusion

HSP-SAM represents a significant step toward fully automated medical image analysis. By replacing manual prompts with a hierarchical, self-prompting mechanism, the framework achieves high accuracy and superior generalization. Future work may involve extending this 2D framework to 3D volumetric data (CT/MRI) and exploring the integration of linguistic representations to allow clinicians to define new tasks through natural language.

Summary updated

Abstract

Although the Segment Anything Model (SAM) is highly effective in natural image segmentation, it requires dependencies on prompts, which limits its applicability to medical imaging where manual prompts are often unavailable. Existing efforts to fine-tune SAM for medical segmentation typically struggle to remove this dependency. We propose Hierarchical Self-Prompting SAM (HSP-SAM), a novel self-prompting framework that enables SAM to achieve strong performance in prompt-free medical image segmentation. Unlike previous self-prompting methods that remain limited to positional prompts similar to vanilla SAM, we are the first to introduce learning abstract prompts during the self-prompting process. This simple and intuitive self-prompting framework achieves superior performance on classic segmentation tasks such as polyp and skin lesion segmentation, while maintaining robustness across diverse medical imaging modalities. Furthermore, it exhibits strong generalization to unseen datasets, achieving improvements of up to 14.04% over previous state-of-the-art methods on some challenging benchmarks. These results suggest that abstract prompts encapsulate richer and higher-dimensional semantic information compared to positional prompts, thereby enhancing the model's robustness and generalization performance. All models and codes will be released upon acceptance.
BibTeX
@misc{zhang2025hierarchical,
  author = {Mengmeng Zhang and Xingyuan Dai and Yicheng Sun and Jing Wang and Yueyang Yao and Xiaoyan Gong and Fu'ze Cong and Fei-Yue Wang and Yisheng Lv},
  title = {{Hierarchical Self-Prompting SAM: A Prompt-Free Medical Image Segmentation Framework}},
  year = {2025},
  eprint = {2506.02854},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.02854},
}
Figure