CholecTrack20: A Dataset for Multi-Class Multiple Tool Tracking in Laparoscopic Surgery

Chinedu Innocent Nwoye, Kareem Elgohary, Anvita Srinivas, Fauzan Zaid, Joël L. Lavanchy, Nicolas PadoyarXiv 2023 · arXiv · OpenReview

Abstract

Tool tracking in surgical videos is vital in computer-assisted intervention for tasks like surgeon skill assessment, safety zone estimation, and human-machine collaboration during minimally invasive procedures. The lack of large-scale datasets hampers Artificial Intelligence implementation in this domain. Current datasets exhibit overly generic tracking formalization, often lacking surgical context: a deficiency that becomes evident when tools move out of the camera's scope, resulting in rigid trajectories that hinder realistic surgical representation. This paper addresses the need for a more precise and adaptable tracking formalization tailored to the intricacies of endoscopic procedures by introducing CholecTrack20, an extensive dataset meticulously annotated for multi-class multi-tool tracking across three perspectives representing the various ways of considering the temporal duration of a tool trajectory: (1) intraoperative, (2) intracorporeal, and (3) visibility within the camera's scope. The dataset comprises 20 laparoscopic videos with over 35,000 frames and 65,000 annotated tool instances with details on spatial location, category, identity, operator, phase, and surgical visual conditions. This detailed dataset caters to the evolving assistive requirements within a procedure.

Introduction

Computer-assisted intervention (CAI) relies heavily on the ability to track surgical tools in real-time. This capability is foundational for tasks such as safety zone estimation, surgeon skill assessment, and human-machine collaboration. However, the development of robust Artificial Intelligence (AI) models for this domain has been hindered by a lack of large-scale, context-aware datasets. Generic Multi-Object Tracking (MOT) formalizations often fail to capture the unique constraints of the surgical environment, such as tools moving out of the camera's field of view (FOV) while remaining active within the patient.

Multi-Perspective Tracking Formalization

To address these limitations, CholecTrack20 introduces a novel tracking formalization based on three distinct temporal perspectives:

  1. Visibility Trajectory: Tracks tools only while they are visible within the camera's FOV. This is critical for real-time feedback.
  2. Intracorporeal Trajectory: Tracks tools from entry to exit of the surgical port (trocar), including periods where they are out of the camera's view (OOCV). This helps in evaluating specific surgical tasks.
  3. Intraoperative Trajectory: A "lifelong" track for each tool instance throughout the entire procedure, requiring re-identification even after a tool has been removed and re-inserted.
Overview of the surgical environment and the multi-perspective tracking framework.

A key innovation in this dataset is the inclusion of operator identity. By linking tools to specific operators (e.g., Main Surgeon's Right Hand), the dataset provides spatial constraints that can significantly reduce identity switches between identical-looking tools, as each tool enters through a fixed trocar location.

The CholecTrack20 Dataset

CholecTrack20 is derived from the Cholec80 and CholecT50 repositories, comprising 20 full-length laparoscopic cholecystectomy videos (over 14 hours of footage). The dataset includes over 35,000 frames annotated at 1 fps, totaling 65,000 tool instances.

Distribution of surgical tool categories and the prevalence of various surgical visual challenges within the dataset.

Annotations are highly granular, covering spatial locations, seven tool categories (grasper, bipolar, hook, scissors, specimen bag, irrigator, clipper), unique track IDs, operator identities, surgical phases, and eight types of visual challenges such as smoke, bleeding, and lens fouling. This level of detail is essential for training models to handle the chaotic and high-stakes environment of the operating room.

Benchmarking and Results

The authors evaluated state-of-the-art (SOTA) detection and tracking models on CholecTrack20. YOLOv7 achieved the best detection performance with an Average Precision (AP) of 56.1%, showing robustness in "bleeding" and "smoke" scenarios. For tracking, Bot-SORT led with a Higher-Order Tracking Accuracy (HOTA) of 44.7% for visibility tracking.

A significant finding is the performance gap between visibility and intracorporeal tracking. Models struggle to maintain identity when tools move off-camera, as visual cues are lost. Furthermore, tracking difficulty varies by surgical phase; for instance, "gallbladder dissection" presents high complexity due to frequent tool interactions and obscured views, whereas "clipping and cutting" is more linear and easier to track.

Conclusion

CholecTrack20 provides a rigorous benchmark for surgical tool tracking, highlighting the limitations of current AI models in handling complex, context-dependent scenarios. The introduction of multi-perspective trajectories and operator-linked tracking offers a path forward for developing more reliable surgical assistants. Future research should focus on incorporating "surgical grammar"—the logical flow of a procedure—to improve re-identification and tracking robustness in challenging environments.

Summary updated

Abstract

Tool tracking in surgical videos is vital in computer-assisted intervention for tasks like surgeon skill assessment, safety zone estimation, and human-machine collaboration during minimally invasive procedures. The lack of large-scale datasets hampers Artificial Intelligence implementation in this domain. Current datasets exhibit overly generic tracking formalization, often lacking surgical context: a deficiency that becomes evident when tools move out of the camera's scope, resulting in rigid trajectories that hinder realistic surgical representation. This paper addresses the need for a more precise and adaptable tracking formalization tailored to the intricacies of endoscopic procedures by introducing CholecTrack20, an extensive dataset meticulously annotated for multi-class multi-tool tracking across three perspectives representing the various ways of considering the temporal duration of a tool trajectory: (1) intraoperative, (2) intracorporeal, and (3) visibility within the camera's scope. The dataset comprises 20 laparoscopic videos with over 35,000 frames and 65,000 annotated tool instances with details on spatial location, category, identity, operator, phase, and surgical visual conditions. This detailed dataset caters to the evolving assistive requirements within a procedure.
BibTeX
@misc{nwoye2023cholectrack,
  author = {Chinedu Innocent Nwoye and Kareem Elgohary and Anvita Srinivas and Fauzan Zaid and Joël L. Lavanchy and Nicolas Padoy},
  title = {{CholecTrack20: A Dataset for Multi-Class Multiple Tool Tracking in Laparoscopic Surgery}},
  year = {2023},
  eprint = {2312.07352},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.07352},
}
Figure