Abstract
Surgical tool detection is a fundamental task for understanding egocentric open surgery videos. However, detecting surgical tools presents significant challenges due to their highly imbalanced class distribution, similar shapes and textures, and heavy occlusion. The lack of a comprehensive large-scale dataset compounds these challenges. In this paper, we introduce EgoSurgery-Tool, an extension of the existing EgoSurgery-Phase dataset, which contains real open surgery videos captured using an egocentric camera attached to the surgeon's head. EgoSurgery-Tool has been densely annotated with over 49K surgical tool bounding boxes across 15 categories and over 46K hand-bounding boxes. We conduct a comprehensive analysis using nine popular object detectors to assess their effectiveness in both surgical tool and hand detection.
Introduction
While AI for Minimally Invasive Surgery (MIS)—like laparoscopy—has seen significant progress due to stable camera views and limited tool sets, open surgery remains a significant challenge for computer vision. In open surgery, the environment is chaotic: surgeons use a vast array of similar-looking tools, hands frequently overlap, and heavy occlusions are the norm. Existing datasets for open surgery are often too small, not publicly available, or lack the egocentric perspective necessary for real-time assistant systems. Furthermore, general-purpose hand detection models fail in the operating room because they are not prepared for the specific textures of surgical gloves and the intense lighting of a surgical suite.
The EgoSurgery-Tool Dataset
EgoSurgery-Tool is an extension of the EgoSurgery-Phase dataset, featuring dense manual annotations on 15,437 high-quality frames from 15 different surgical videos. It offers significantly higher density than typical MIS datasets, with 3.2 tool instances per frame.
Key features include:
- 15 Tool Categories: Ranging from common items like Gauze and Tweezers to specialized instruments like Raspatories and Mouth Gags.
- 4 Hand Categories: Distinguishing between the "own" (surgeon's) left/right hands and "other" (assistants') left/right hands.
Tool-Phase Correlation and Hand Detection
A key insight from the dataset is the strong correlation between specific tools and surgical phases. For example, a "Skewer" has a 100% probability of appearing during the "Design" phase, while a "Scalpel" is a 98% indicator of the "Incision" phase. This suggests that tool detection is a powerful proxy for broader surgical workflow understanding.
Additionally, the researchers found a significant domain gap. Models trained on general-activity datasets like EgoHands achieved only 8.9 AP on surgical hand detection. Training on EgoSurgery-Tool increased performance to 55.3 AP, highlighting the necessity of domain-specific surgical data.
Benchmarking and Results
The researchers evaluated nine state-of-the-art object detection architectures, including Faster R-CNN, VarifocalNet, and DINO. VarifocalNet emerged as the top performer:
- Surgical Tool Detection: VarifocalNet achieved an AP of 45.8, outperforming DINO (39.7) and Faster R-CNN (37.7). Its success is attributed to its ability to handle dense objects and heavy occlusion.
- Hand Detection: VarifocalNet led with 59.4 AP, though DINO showed superior precision for high-overlap scenarios ($AP_{75}$ of 65.6).
Conclusion
EgoSurgery-Tool provides a much-needed benchmark for developing AI assistants in complex surgical environments. While the dataset addresses scale and variety, challenges remain regarding extreme class imbalance and visual similarity between metallic tools. Future work will likely focus on temporal modeling to improve detection under heavy occlusion by utilizing video context over time.
Summary updated