SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation

Jieming Yu, An Wang, Wenzhen Dong, Mengya Xu, Mobarakol Islam, Jie Wang, Long Bai, Hongliang Ren

Abstract

This paper presents the first comprehensive empirical evaluation of Meta’s Segment Anything Model 2 (SAM 2) for surgical instrument segmentation. By leveraging a new memory mechanism and mask decoder, SAM 2 addresses the limitations of its predecessor in handling temporal data and domain-specific challenges. Our findings indicate that SAM 2 achieves state-of-the-art (SOTA) zero-shot performance on standard benchmarks, significantly outperforming the original SAM in both accuracy and inference speed while demonstrating remarkable robustness to common surgical image corruptions.

Introduction

Precise surgical instrument segmentation is vital for automated surgical assistance, skill assessment, and intraoperative navigation. Traditional deep learning approaches often require large, meticulously annotated datasets, which are difficult to obtain in the medical field. The introduction of the Segment Anything Model (SAM) provided a powerful zero-shot tool for general computer vision, yet it struggled with the unique visual characteristics of surgical environments and lacked native video support. SAM 2 aims to bridge this gap with a unified architecture for real-time promptable object segmentation in both images and videos.

Methodology

The study evaluates SAM 2's performance on two gold-standard benchmarks: MICCAI EndoVis 2017 and EndoVis 2018. The evaluation focused on "zero-shot" capabilities, testing the model without any fine-tuning on surgical data.

Prompting Strategies

  1. Static Image Prompts: Models were prompted with either a single point (centroid) or a bounding box for each frame.
  2. Video Sequence Prompts: SAM 2 was prompted with a single point only in the initial frame, utilizing its memory mechanism to track the instrument throughout the sequence.

Robustness Testing

To simulate real-world surgical conditions, the researchers subjected the models to 18 types of image corruptions across five levels of severity. These included noise (Gaussian, shot, impulse, speckle), blur (defocus, glass, motion, zoom, Gaussian), weather-like effects (snow, frost, fog), and digital distortions (brightness, contrast, pixelation, JPEG compression, saturation).

Results and Analysis

Quantitative Performance

SAM 2 demonstrated a significant performance leap over SAM 1 and other specialized models. With bounding box prompts, SAM 2 achieved a Binary IoU of 90.97 on EndoVis 2017, surpassing existing SOTA models specifically trained on surgical data.

Table 1: Performance comparison of SAM 2 against SAM and other specialized surgical segmentation models on EndoVis 2017 and 2018 datasets.

Notably, SAM 2's point-prompt performance improved by 20–30% compared to SAM 1. The memory mechanism proved highly effective; SAM 2 prompted only in the first frame often outperformed SAM 1 even when the latter was prompted in every frame.

Inference Speed

Efficiency is critical for clinical adoption. SAM 2 achieved an inference speed of approximately 8.8 frames per second (FPS) on an NVIDIA RTX 3090, representing a 2.9x speedup over the original SAM (~3.0 FPS).

Robustness to Corruptions

Surgical environments are prone to visual artifacts like smoke (fog), lighting changes, and motion blur. SAM 2 exhibited superior resilience compared to SAM 1 across nearly all corruption categories.

Figure 1: Visual comparison of SAM and SAM 2 performance under various image corruptions, demonstrating SAM 2's enhanced robustness.

Even at the highest severity levels, SAM 2 maintained functional segmentation masks where previous models failed, particularly under conditions of varying brightness and simulated surgical smoke.

Discussion and Conclusion

SAM 2 represents a major advancement for zero-shot surgical video segmentation. Its ability to maintain temporal consistency through a memory mechanism, combined with its increased speed and robustness, makes it a promising candidate for real-time clinical applications.

However, limitations remain. The model occasionally struggles with the precise delineation of fine metallic edges and very small instrument components. Future research should focus on integrating automated prompting systems and exploring text-to-segmentation capabilities to further reduce the need for manual intervention in the operating room.

Reference: Yu, J., et al. "SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation." arXiv:2408.04593v1, 2024.

Summary updated

BibTeX
@inproceedings{yu2025sam,
  author = {Jieming Yu and An Wang and Wenzhen Dong and Mengya Xu and Mobarakol Islam and Jie Wang and Long Bai and Hongliang Ren},
  title = {{SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation}},
  booktitle = {EMA4MICCAI},
  year = {2025},
  doi = {10.1007/978-3-032-13961-0\_18},
}
Figure