Abstract
The Segment Anything Model (SAM) can achieve satisfactory segmentation performance under high-quality box prompts. However, SAM's robustness is compromised by the decline in box quality, limiting its practicality in clinical reality. In this study, we propose a novel Robust Box prompt based SAM (RoBox-SAM) to ensure SAM's segmentation performance under prompts with different qualities. Our contribution is three-fold. First, we propose a prompt refinement module to implicitly perceive the potential targets, and output the offsets to directly transform the low-quality box prompt into a high-quality one. We then provide an online iterative strategy for further prompt refinement. Second, we introduce a prompt enhancement module to automatically generate point prompts to assist the box-promptable segmentation effectively. Last, we build a self-information extractor to encode the prior information from the input image. These features can optimize the image embeddings and attention calculation, thus, the robustness of SAM can be further enhanced. Extensive experiments on the large medical segmentation dataset including 99,299 images, 5 modalities, and 25 organs/targets validated the efficacy of our proposed RoBox-SAM.
Introduction
The Segment Anything Model (SAM) has revolutionized image segmentation by providing a foundation model capable of zero-shot generalization. However, in medical imaging, SAM exhibits a significant "glass jaw": extreme sensitivity to the quality of user-provided prompts. While "tight" bounding boxes yield excellent results, clinical practice often involves "low-quality" prompts—boxes that are oversized, off-center, or shifted due to the fast-paced nature of medical workflows.
Imprecise prompts can lead to a drastic drop in performance. For instance, a 20-30% random shift in a box prompt can cause SAM's performance on CT scans to plummet by nearly 50%. This "attention drift" makes standard SAM unreliable for high-stakes clinical applications. To address this, we present RoBox-SAM, a specialized framework designed to handle imprecise user inputs by automatically refining prompts and enhancing image features.
RoBox-SAM Architecture
RoBox-SAM integrates three lightweight modules into the SAM architecture to rectify prompts before final segmentation. Crucially, the heavy SAM backbone (ViT-H) remains frozen, while only the new modules and the mask decoder are fine-tuned, ensuring computational efficiency.
- Prompt Refinement Module (PRM): Acting as a "box corrector," the PRM utilizes image features to predict coordinate offsets ($\Delta x, \Delta y$). It "nudges" the corners of a low-quality box to better encapsulate the target organ. This process can be applied iteratively to progressively home in on the object.
- Prompt Enhancement Module (PEM): The PEM identifies areas between the original "noisy" box and the refined box as likely background. It automatically generates "negative" point prompts in these regions, instructing the model to exclude these areas from the final mask.
- Self-Information Extractor (SIE): To better capture medical textures, the SIE extracts prior information such as Canny edges, SLIC superpixels, and frequency-domain data (via Fast Fourier Transform). These features are fused with SAM’s image embeddings to sharpen the model's focus on anatomical boundaries.
Experimental Setup
The efficacy of RoBox-SAM was validated on a massive dataset comprising 99,299 images across five modalities: CT, MRI, Ultrasound, Colonoscopy, and Fundus imaging. The study covered 25 different targets, including heart structures, brain tumors, breast nodules, and polyps. Data sources included 14 public datasets and 5 private ultrasound datasets annotated by expert sonographers.
Results and Discussion
RoBox-SAM significantly outperforms standard SAM and medical-specific variants like MedSAM and HQ-SAM, particularly as prompt quality degrades.
Robustness to Noise
In CT scans with 20-30% prompt noise, MedSAM’s DICE score dropped to 49.98%, whereas RoBox-SAM maintained a robust 84.19%. The model's Prompt Robustness (PR) score—measuring output consistency despite prompt shifts—was consistently higher across all modalities.
Key Insights
- Implicit Perception: By first understanding where a box should have been, the model mitigates the impact of human error.
- The Power of Negative Space: Utilizing "negative space" between refined and original boxes prevents over-segmentation in noisy environments.
- Frequency Domain Matters: By using Fourier Transforms to highlight edges and details, the model becomes much more robust to the "speckle noise" common in ultrasound images.
Conclusion
RoBox-SAM provides a robust solution for medical image segmentation by making SAM resilient to imprecise user inputs. While currently optimized for 2D slices, future work will focus on extending this framework to native 3D segmentation tasks and a broader range of clinical imaging modalities. The addition of these modules is highly efficient, adding less than 1 million parameters and only ~0.6 seconds of processing time per image.
Summary updated