Seeing Through the Tool: A Controlled Benchmark for Occlusion Robustness in Foundation Segmentation Models

Nhan Ho, Luu Le, Thanh-Huy Nguyen, Thien Nguyen, Xiaofeng Liu, Ulas BagciarXiv 2026 · arXiv

TL;DR

The paper introduces OccSAM-Bench, a new benchmark to evaluate how Segment Anything Model (SAM) variants handle surgical tool occlusions in clinical endoscopy. By breaking down evaluation into visible, invisible, and full target regions, the authors reveal that standard metrics hide dangerous model behaviors, splitting foundation models into two distinct archetypes: those that safely reject instruments and those that attempt to "see through" them.

What Problem Does This Paper Address?

In real-world surgery, target anatomy (like a polyp) is frequently blocked by moving surgical instruments. While foundation models like SAM have shown incredible promise in medical image segmentation, they are almost exclusively tested on clean, perfectly curated datasets.

When these models face occlusions, standard evaluation metrics fall short. Typically, researchers look at "full mask" overlap. However, in a surgical setting, if a model blindly predicts that target tissue exists on top of a metallic surgical tool, it might score highly on a standard metric by overlapping the hidden anatomy. In practice, this is a hazardous false positive that could misguide robotic navigation or surgical planning. This paper addresses the lack of a systematic, safety-aligned framework to evaluate how foundation models behave when their sight is blocked.

Figure 1. OccSAM-Bench: evaluating segmentation under sur- gical occlusion. Clinical endoscopy often involves partial occlu- sion of target anatomy by surgical instruments. OccSAM-Bench simulates such conditions by overlaying surgical tools onto polyp images to create controlled occlusions. Models are evaluated using a three-region protocol that separates visible, invisible (occluded), and full masks, revealing behaviors that standard full-mask evalu- ation alone cannot capture.

Methods

The authors developed OccSAM-Bench using two main components:

  1. Controlled Occlusion Generation: To isolate different types of visual challenges, the framework generates two types of synthetic occlusions at three calibrated severity levels (Low, Medium, High):
  2. Surgical Tool Paste: Real instrument masks from the Kvasir-Instrument dataset are scaled, rotated, and overlaid onto target images.
  3. Cutout Occlusion: Rectangular patches of the target are removed entirely, isolating the effect of missing data without introducing a foreign object.
  4. Three-Region Evaluation Protocol: Instead of relying on a single full-mask score, the authors split the ground truth into three distinct targets:
  5. Visible-only ($M_{vis}$): The tissue actually seen. Models are penalized if they bleed into the tool area.
  6. Invisible ($M_{inv}$): The hidden tissue behind the tool.
  7. Full ($M_{full}$): The complete amodal target.

The models were tested using both bounding box prompts and single interior point prompts.

Datasets

The benchmark was constructed using three widely accepted public polyp segmentation datasets:

  • CVC-300 (60 images)
  • CVC-ColonDB (380 images)
  • ETIS-LaribPolypDB (196 images)

Seven models were evaluated in a zero-shot manner (without any dataset-specific fine-tuning): general-purpose models (SAM, SAM 2, SAM 3) and medically adapted ones (MedSAM, SAM-Med2D, MedSAM2, MedSAM3).

Key Insights and Ideas

  • The Full-Mask Trap: Standard full-mask evaluation is an artifact-prone metric. Models like MedSAM appeared robust on full-mask scores but actually suffered catastrophic performance drops when evaluating the visible-only region, meaning they were simply guessing blindly through the tool.
  • Two Behavioral Archetypes: The paper uncovers a clear architectural split under occlusion:
  • Occluder-Aware models (SAM, SAM 2, SAM 3, MedSAM3) treat the tool as background and strictly delineate visible tissue.
  • Occluder-Agnostic models (MedSAM, MedSAM2) confidently predict through the tool, attempting amodal completion.
  • The Overfitting Cost of Medical Fine-Tuning: Medically adapted models built on SAM 1 and 2 (like MedSAM and SAM-Med2D) exhibited severe performance collapse under high occlusion, degrading by 41–60%. Because they were fine-tuned on pristine medical images, the sharp edges of surgical tools acted like out-of-distribution adversarial perturbations.
Figure 2. OccSAM-Bench overview. Left: Surgical-tool occlusions are simulated by pasting instrument masks onto polyp images from three colonoscopy datasets. Right: SAM-family models exhibit two behaviors under occlusion: Occluder-Aware models segment visible tissue while rejecting the instrument, whereas Occluder-Agnostic models predict through the occluder. Both can achieve similar full-mask scores, showing that standard evaluation obscures these differences.

Key Results

  • SAM 3 emerged as the most robust model for conservative, visible-tissue segmentation, holding the top position at high severity across all three datasets (achieving visible Dice Similarity Coefficients [DSC] of 0.724, 0.748, and 0.801).
  • MedSAM2 stood out as a unique boundary case. It was the only model capable of true, boundary-consistent amodal completion—maintaining a competitive visible DSC while scoring high on the invisible region.
  • Prompt Sensitivity: Bounding box prompts consistently outperformed single-point prompts under occlusion. Notably, MedSAM completely collapsed under single-point prompting (near-zero DSC) because its prompt encoder was exclusively fine-tuned on bounding boxes.

Limitations and Future Work

The primary limitation is that OccSAM-Bench relies on 2D RGB images and simplified synthetic overlays. This abstraction bypasses complex, real-world surgical physics such as specular reflections, dynamic lighting changes, shadows, and tool-induced tissue deformation.

The authors note that future work must extend this paradigm to video-based temporal tracking, physically realistic 3D rendering, and multi-modality surgical datasets.

Summary updated

Abstract

Occlusion, where target structures are partially hidden by surgical instruments or overlapping tissues, remains a critical yet underexplored challenge for foundation segmentation models in clinical endoscopy. We introduce OccSAM-Bench, a benchmark designed to systematically evaluate SAM-family models under controlled, synthesized surgical occlusion. Our framework simulates two occlusion types (i.e., surgical tool overlay and cutout) across three calibrated severity levels on three public polyp datasets. We propose a novel three-region evaluation protocol that decomposes segmentation performance into full, visible-only, and invisible targets. This metric exposes behaviors that standard amodal evaluation obscures, revealing two distinct model archetypes: Occluder-Aware models (SAM, SAM 2, SAM 3, MedSAM3), which prioritize visible tissue delineation and reject instruments, and Occluder-Agnostic models (MedSAM, MedSAM2), which confidently predict into occluded regions. SAM-Med2D aligns with neither and underperforms across all conditions. Ultimately, our results demonstrate that occlusion robustness is not uniform across architectures, and model selection must be driven by specific clinical intent-whether prioritizing conservative visible-tissue segmentation or the amodal inference of hidden anatomy.
BibTeX
@misc{ho2026seeing,
  author = {Nhan Ho and Luu Le and Thanh-Huy Nguyen and Thien Nguyen and Xiaofeng Liu and Ulas Bagci},
  title = {{Seeing Through the Tool: A Controlled Benchmark for Occlusion Robustness in Foundation Segmentation Models}},
  year = {2026},
  eprint = {2604.11711},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.11711},
}

Loading related content...

Figure