Abstract
Surgical computer vision has long been constrained by the "closed-set" nature of existing models, which are limited to identifying a small number of pre-defined classes due to the high cost of expert manual annotation. We introduce RASO (Recognize Any Surgical Object), a foundation model designed for open-set surgical object recognition in both images and videos. By leveraging a novel weakly-supervised data pipeline that extracts 3.6 million tags from unannotated surgical lecture videos, RASO bypasses the manual labeling bottleneck. The model achieves state-of-the-art zero-shot and supervised performance across multiple benchmarks while achieving a 4x increase in video processing speed through a custom temporal-attention fusion layer.
Introduction
In the operating room, computer vision models must perform two primary tasks: localization (segmenting object boundaries) and recognition (identifying the object or action). While general-domain models like the Segment Anything Model (SAM) have demonstrated exceptional localization capabilities, they lack the domain-specific vocabulary required to identify surgical tools, anatomical structures, or complex procedural actions without human intervention.
Furthermore, existing surgical recognition models are typically "closed-set," restricted to 10–20 classes because manual annotation requires expensive medical expertise. Additionally, many models process video frame-by-frame, failing to capture the temporal context necessary to understand dynamic surgical actions. RASO addresses these challenges by utilizing large-scale, weakly-supervised data to build an open-set recognition system capable of understanding the nuances of surgery in real-time.
The Weakly-Supervised Data Pipeline
To solve the data scarcity problem, the authors developed a Weakly-Supervised Data Generation Pipeline that transforms existing educational resources into training data. By scraping 2,200 surgical lecture videos from WebSurg, the pipeline utilizes several automated components:
- Transcription and Extraction: WhisperX transcribes educational voiceovers, while spaCy identifies surgical entities such as organs and tools.
- Action Parsing: A SceneGraphParser extracts action triplets (e.g.,
<grasper, dissects, gallbladder>), providing semantic context to the visual data. - LLM Augmentation: GPT-4o was employed to auto-annotate an additional 120,000 images, further diversifying the training set.
This process resulted in a massive pretraining dataset containing 901,000 images, 1.2 million sentences, and 3.6 million tag annotations across 2,066 unique surgical tags.
Model Architecture and Temporal Fusion
RASO is built upon the Recognize Anything Model (RAM) framework, utilizing a Swin-Transformer for image encoding and a frozen CLIP text encoder for tag processing.
A key innovation is the Temporal-Attention Fusion Layer. Unlike standard models that treat video as a sequence of independent images, RASO stacks frame-level features along a temporal axis. A multi-head attention mechanism then aggregates these dependencies before decoding the tags. This allows the model to differentiate between static objects (nouns) and dynamic actions (verbs), which are inherently time-dependent.
Experimental Results
RASO was evaluated on four standard surgical benchmarks: CholecT50 (action triplets), Cholec80 (phases and tools), GraSP (multi-level understanding), and RSS (robotic segmentation).
Zero-Shot and Supervised Performance
Without any task-specific manual annotations, RASO significantly outperformed existing medical vision-language models like SurgVLP and BiomedCLIP. It achieved mAP improvements of 2.9 on CholecT50, 4.5 on Cholec80, 10.6 on RSS, and 7.2 on GraSP.
In supervised settings, RASO established a new state-of-the-art on the CholecT50 action triplet task with a 57.5 mAP, surpassing the previous leading model, Rendezvous (56.4 mAP). Notably, the model's pretraining efficiency allowed it to reach these results in just 4 epochs, compared to the 200+ epochs required by traditional models.
Video Efficiency
The temporal-attention fusion layer provided both accuracy and speed benefits. The video-optimized version of RASO achieved a higher mAP (61.3%) than a naive frame-by-frame approach (58.8%). More importantly, it is approximately 4x faster, reducing latency from 398.8 ms to 100.8 ms per batch, making it highly suitable for real-time applications.
Discussion and Future Work
While the weakly-supervised approach allows for unprecedented scale, it introduces noise. GPT-4o annotations showed an agreement rate with human clinicians ranging from 19% to 35% for frequent tags. However, the sheer volume of data appears to allow the model to learn robust features despite this noise.
RASO serves as a foundational step toward "grounded surgical segmentation." By integrating RASO’s open-set recognition with localization models like SAM, future systems could automatically generate mask-category pairs in real-time for robotic-assisted surgeries, eliminating the need for manual human prompting and paving the way for more autonomous surgical assistance.
Summary updated