A surgical activity model of laparoscopic cholecystectomy for co-operation with collaborative robots

R. T. Younis, Amine Yamlahi, Sebastian Bodenstedt, Pm. Scheikl, Anna Kisilenko, Marcus Daum, A. Schulze, Philipp Anthony Wise, Felix Nickel, Franziska Mathis-Ullrich, Lena Maier‐Hein, B-p. Muller-Stich, Stefanie Speidel, Marius Distler, Jürgen Weitz, Martin WagnerSurgical 2024 · DOI

Introduction

Laparoscopic cholecystectomy is a cornerstone of modern surgical practice. However, the increasing demand for surgical procedures, coupled with a global shortage of medical personnel, necessitates the development of collaborative surgical robots (cobots). Unlike teleoperated systems, cobots are designed to work alongside surgeons autonomously, requiring a high degree of context-awareness to anticipate and execute tasks. This study presents a fine-grained activity model based on "quadruplets" to enable such robotic collaboration.

Graphical abstract summarizing the study workflow, machine learning approach, and key automation findings.

Methodology: The Quadruplet Model

To provide robots with the necessary context, the researchers developed a model that extends beyond simple action triplets. The proposed quadruplet model consists of:

  1. Actor: Identifying the specific hand involved (e.g., Surgeon's left hand vs. Assistant's right hand).
  2. Instrument: The tool being utilized (e.g., Grasper, Clipper).
  3. Action: The specific verb or task (e.g., Hold, Cut, Cauterize).
  4. Target: The anatomical structure or object being manipulated (e.g., Gallbladder, Liver).

This granular approach allows for a precise mapping of surgical workflow, distinguishing between the roles of the primary surgeon and the assistant.

Sankey diagram illustrating the distribution and flow of activity quadruplets from actors to targets.

Data and Machine Learning

The study introduced the HeiCholeActivity dataset, comprising 20 annotated videos (11.7 hours) of both human and ex-vivo porcine procedures. This dataset is unique for its inclusion of "Actor" labels and its bridge between preclinical animal models and clinical human data. For activity recognition, the researchers employed Distilled-Swin, a Swin Transformer-based model. The model was trained on a combination of HeiCholeActivity and the benchmark CholecT45 dataset to improve generalizability and performance in the face of significant class imbalance.

Key Results and Automation Potential

The machine learning model achieved a mean Average Precision (mAP) of 37.9% and a Top-5 accuracy of 85.3%. While recognizing infrequent "edge case" activities remains a challenge, the model demonstrated robust performance on frequent surgical tasks.

A critical finding of the activity analysis is the quantification of robotic potential. The study revealed that up to 83.5% of an assistant's tissue-interacting tasks (excluding camera control) consist of repetitive "grasp" and "hold" actions. This suggests that even low-level robotic autonomy could significantly reduce the physical and cognitive workload of surgical assistants.

Conclusion

The development of a standardized quadruplet model and the HeiCholeActivity dataset marks a significant step toward autonomous surgical collaboration. By identifying that the vast majority of assistant tasks are candidates for automation, this research provides a roadmap for the deployment of cobots in the operating room. Future work will focus on integrating multi-modal data and scaling datasets to overcome current limitations in recognizing rare surgical events.

Summary updated

Abstract

An activity model was developed and applied for the fine-granular annotation of laparoscopic cholecystectomies in two surgical settings. A machine recognition algorithm trained on our own annotated dataset and CholecT45 achieved a higher performance than training only on CholecT45 and can recognize frequently occurring activities well, but not infrequent activities. The analysis of an annotated dataset allowed for the quantification of the potential of collaborative surgical robots to address the workload of surgical staff. If collaborative surgical robots could grasp and hold tissue, up to 83.5% of the assistant's tissue interacting tasks (i.e. excluding camera guidance) could be performed by robots.
BibTeX
@inproceedings{younis2024surgical,
  author = {R. T. Younis and Amine Yamlahi and Sebastian Bodenstedt and PM. Scheikl and Anna Kisilenko and Marcus Daum and A. Schulze and Philipp Anthony Wise and Felix Nickel and Franziska Mathis-Ullrich and Lena Maier‐Hein and B-P. Muller-Stich and Stefanie Speidel and Marius Distler and Jürgen Weitz and Martin Wagner},
  title = {{A surgical activity model of laparoscopic cholecystectomy for co-operation with collaborative robots}},
  booktitle = {Surgical},
  year = {2024},
  doi = {10.1007/s00464-024-10958-w},
}
Figure