TL;DR
The paper introduces OccSAM-Bench, a new benchmark to evaluate how Segment Anything Model (SAM) variants handle surgical tool occlusions in clinical endoscopy. By breaking down evaluation into visible, invisible, and full target regions, the authors reveal that standard metrics hide dangerous model behaviors, splitting foundation models into two distinct archetypes: those that safely reject instruments and those that attempt to "see through" them.
What Problem Does This Paper Address?
In real-world surgery, target anatomy (like a polyp) is frequently blocked by moving surgical instruments. While foundation models like SAM have shown incredible promise in medical image segmentation, they are almost exclusively tested on clean, perfectly curated datasets.
When these models face occlusions, standard evaluation metrics fall short. Typically, researchers look at "full mask" overlap. However, in a surgical setting, if a model blindly predicts that target tissue exists on top of a metallic surgical tool, it might score highly on a standard metric by overlapping the hidden anatomy. In practice, this is a hazardous false positive that could misguide robotic navigation or surgical planning. This paper addresses the lack of a systematic, safety-aligned framework to evaluate how foundation models behave when their sight is blocked.
Methods
The authors developed OccSAM-Bench using two main components:
- Controlled Occlusion Generation: To isolate different types of visual challenges, the framework generates two types of synthetic occlusions at three calibrated severity levels (Low, Medium, High):
- Surgical Tool Paste: Real instrument masks from the Kvasir-Instrument dataset are scaled, rotated, and overlaid onto target images.
- Cutout Occlusion: Rectangular patches of the target are removed entirely, isolating the effect of missing data without introducing a foreign object.
- Three-Region Evaluation Protocol: Instead of relying on a single full-mask score, the authors split the ground truth into three distinct targets:
- Visible-only ($M_{vis}$): The tissue actually seen. Models are penalized if they bleed into the tool area.
- Invisible ($M_{inv}$): The hidden tissue behind the tool.
- Full ($M_{full}$): The complete amodal target.
The models were tested using both bounding box prompts and single interior point prompts.
Datasets
The benchmark was constructed using three widely accepted public polyp segmentation datasets:
- CVC-300 (60 images)
- CVC-ColonDB (380 images)
- ETIS-LaribPolypDB (196 images)
Seven models were evaluated in a zero-shot manner (without any dataset-specific fine-tuning): general-purpose models (SAM, SAM 2, SAM 3) and medically adapted ones (MedSAM, SAM-Med2D, MedSAM2, MedSAM3).
Key Insights and Ideas
- The Full-Mask Trap: Standard full-mask evaluation is an artifact-prone metric. Models like MedSAM appeared robust on full-mask scores but actually suffered catastrophic performance drops when evaluating the visible-only region, meaning they were simply guessing blindly through the tool.
- Two Behavioral Archetypes: The paper uncovers a clear architectural split under occlusion:
- Occluder-Aware models (SAM, SAM 2, SAM 3, MedSAM3) treat the tool as background and strictly delineate visible tissue.
- Occluder-Agnostic models (MedSAM, MedSAM2) confidently predict through the tool, attempting amodal completion.
- The Overfitting Cost of Medical Fine-Tuning: Medically adapted models built on SAM 1 and 2 (like MedSAM and SAM-Med2D) exhibited severe performance collapse under high occlusion, degrading by 41–60%. Because they were fine-tuned on pristine medical images, the sharp edges of surgical tools acted like out-of-distribution adversarial perturbations.
Key Results
- SAM 3 emerged as the most robust model for conservative, visible-tissue segmentation, holding the top position at high severity across all three datasets (achieving visible Dice Similarity Coefficients [DSC] of 0.724, 0.748, and 0.801).
- MedSAM2 stood out as a unique boundary case. It was the only model capable of true, boundary-consistent amodal completion—maintaining a competitive visible DSC while scoring high on the invisible region.
- Prompt Sensitivity: Bounding box prompts consistently outperformed single-point prompts under occlusion. Notably, MedSAM completely collapsed under single-point prompting (near-zero DSC) because its prompt encoder was exclusively fine-tuned on bounding boxes.
Limitations and Future Work
The primary limitation is that OccSAM-Bench relies on 2D RGB images and simplified synthetic overlays. This abstraction bypasses complex, real-world surgical physics such as specular reflections, dynamic lighting changes, shadows, and tool-induced tissue deformation.
The authors note that future work must extend this paradigm to video-based temporal tracking, physically realistic 3D rendering, and multi-modality surgical datasets.
Summary updated