EgoSurgery-Tool: A Dataset of Surgical Tool and Hand Detection from Egocentric Open Surgery Videos

Ryo Fujii, Hideo Saito, Hiroki Kajita

Abstract

Surgical tool detection is a fundamental task for understanding egocentric open surgery videos. However, detecting surgical tools presents significant challenges due to their highly imbalanced class distribution, similar shapes and textures, and heavy occlusion. The lack of a comprehensive large-scale dataset compounds these challenges. In this paper, we introduce EgoSurgery-Tool, an extension of the existing EgoSurgery-Phase dataset, which contains real open surgery videos captured using an egocentric camera attached to the surgeon's head. EgoSurgery-Tool has been densely annotated with over 49K surgical tool bounding boxes across 15 categories and over 46K hand-bounding boxes. We conduct a comprehensive analysis using nine popular object detectors to assess their effectiveness in both surgical tool and hand detection.

Introduction

While AI for Minimally Invasive Surgery (MIS)—like laparoscopy—has seen significant progress due to stable camera views and limited tool sets, open surgery remains a significant challenge for computer vision. In open surgery, the environment is chaotic: surgeons use a vast array of similar-looking tools, hands frequently overlap, and heavy occlusions are the norm. Existing datasets for open surgery are often too small, not publicly available, or lack the egocentric perspective necessary for real-time assistant systems. Furthermore, general-purpose hand detection models fail in the operating room because they are not prepared for the specific textures of surgical gloves and the intense lighting of a surgical suite.

The EgoSurgery-Tool Dataset

EgoSurgery-Tool is an extension of the EgoSurgery-Phase dataset, featuring dense manual annotations on 15,437 high-quality frames from 15 different surgical videos. It offers significantly higher density than typical MIS datasets, with 3.2 tool instances per frame.

Key features include:

  • 15 Tool Categories: Ranging from common items like Gauze and Tweezers to specialized instruments like Raspatories and Mouth Gags.
  • 4 Hand Categories: Distinguishing between the "own" (surgeon's) left/right hands and "other" (assistants') left/right hands.
Distribution of surgical tool categories, highlighting the long-tail challenge.

Tool-Phase Correlation and Hand Detection

A key insight from the dataset is the strong correlation between specific tools and surgical phases. For example, a "Skewer" has a 100% probability of appearing during the "Design" phase, while a "Scalpel" is a 98% indicator of the "Incision" phase. This suggests that tool detection is a powerful proxy for broader surgical workflow understanding.

Co-occurrence matrix showing how specific tools correlate with surgical phases.

Additionally, the researchers found a significant domain gap. Models trained on general-activity datasets like EgoHands achieved only 8.9 AP on surgical hand detection. Training on EgoSurgery-Tool increased performance to 55.3 AP, highlighting the necessity of domain-specific surgical data.

Benchmarking and Results

The researchers evaluated nine state-of-the-art object detection architectures, including Faster R-CNN, VarifocalNet, and DINO. VarifocalNet emerged as the top performer:

  • Surgical Tool Detection: VarifocalNet achieved an AP of 45.8, outperforming DINO (39.7) and Faster R-CNN (37.7). Its success is attributed to its ability to handle dense objects and heavy occlusion.
  • Hand Detection: VarifocalNet led with 59.4 AP, though DINO showed superior precision for high-overlap scenarios ($AP_{75}$ of 65.6).

Conclusion

EgoSurgery-Tool provides a much-needed benchmark for developing AI assistants in complex surgical environments. While the dataset addresses scale and variety, challenges remain regarding extreme class imbalance and visual similarity between metallic tools. Future work will likely focus on temporal modeling to improve detection under heavy occlusion by utilizing video context over time.

Summary updated

Abstract

Surgical tool detection is a fundamental task for understanding egocentric open surgery videos. However, detecting surgical tools presents significant challenges due to their highly imbalanced class distribution, similar shapes and similar textures, and heavy occlusion. The lack of a comprehensive large-scale dataset compounds these challenges. In this paper, we introduce EgoSurgery-Tool, an extension of the existing EgoSurgery-Phase dataset, which contains real open surgery videos captured using an egocentric camera attached to the surgeon's head, along with phase annotations. EgoSurgery-Tool has been densely annotated with surgical tools and comprises over 49K surgical tool bounding boxes across 15 categories, constituting a large-scale surgical tool detection dataset. EgoSurgery-Tool also provides annotations for hand detection with over 46K hand-bounding boxes, capturing hand-object interactions that are crucial for understanding activities in egocentric open surgery. EgoSurgery-Tool is superior to existing datasets due to its larger scale, greater variety of surgical tools, more annotations, and denser scenes. We conduct a comprehensive analysis of EgoSurgery-Tool using nine popular object detectors to assess their effectiveness in both surgical tool and hand detection. The dataset will be released at https://github.com/Fujiry0/EgoSurgery.
BibTeX
@misc{fujii2024egosurgery,
  author = {Ryo Fujii and Hideo Saito and Hiroki Kajita},
  title = {{EgoSurgery-Tool: A Dataset of Surgical Tool and Hand Detection from Egocentric Open Surgery Videos}},
  year = {2024},
  eprint = {2406.03095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.03095},
}
Figure