ClipAssistNet: bringing real-time safety feedback to operating rooms

Florian Aspart, Jon Lindström Bolmgren, Joël L. Lavanchy, Guido Beldi, Michael S. Woods, Nicolas Padoy, Enes HosgorInt. J. Comput. Assist. Radiol. Surg 2022 · OpenReview

TL;DR

This paper introduces ClipAssistNet, a novel deep learning system that provides real-time visual feedback to surgeons during laparoscopic cholecystectomy, specifically alerting them when the critical clip applier tip is not adequately visible. By ensuring the surgeon maintains good visibility of the instrument, the system aims to enhance safety and reduce complications during this crucial surgical step.

What Problem Does This Paper Address?

Laparoscopic surgeries, while offering benefits like faster recovery, present unique challenges due to the limited field of view and the need to manipulate instruments through small incisions. A particularly critical phase in laparoscopic cholecystectomy (a common procedure) involves clipping the cystic duct and artery before cutting them. Inadequate visibility of the clip applier's tip during this maneuver can lead to serious complications, such as bile leaks or unintentional injury to surrounding tissues. Existing surgical skill assessments and instrument tracking methods often focus on detecting the instrument itself or its visible parts, but not specifically on ensuring the visibility of the critical tip for safe operation. This paper addresses the need for an intra-operative safety feedback mechanism that directly monitors and alerts surgeons about poor visibility of the clip applier's tip during this high-stakes procedure.

Methods

The core of the solution is ClipAssistNet, a neural network-based image classifier designed to detect the visibility of the clip applier's tip in individual video frames. The authors employed a ResNet50 architecture, pre-trained on ImageNet, and fine-tuned it for this specific task. To improve robustness and performance, ClipAssistNet is an ensemble of five such ResNet50 models, trained on different subsets of the data. The model classifies frames into two categories: "tip visible" or "tip invisible." The "tip invisible" cases are defined as frames where the tip is not present or is occluded by other tools or tissues. The authors also explored various loss functions (standard BCE, class-weighted BCE, video-weighted BCE, and BCE with label noise correction) to address dataset peculiarities like class imbalance and potential labeling noise.

Datasets

The authors introduced a new dataset specifically for this task, comprising 300 laparoscopic cholecystectomy videos. These videos were frame-wise annotated for clipper tip visibility. The dataset was split into a training/validation set (271 videos) and a test set (29 videos). Notably, the test set was triply annotated to ensure annotation consistency and mitigate labeling noise.

Key Insights and Ideas

  • Visibility as a Safety Proxy: The paper posits that ensuring the visibility of the clip applier's tip is a direct proxy for safe clipping, preventing both misplacement and unintended tissue damage.
  • Context-Independent Annotation: Annotations were made frame-by-frame, independent of surrounding frames, to mimic the real-world scenario where a surgeon must make decisions based on the current view. This also means that frames with poor contrast or motion blur, where visibility is ambiguous, are labeled as "invisible."
  • Ensembling for Robustness: Combining predictions from multiple ResNet50 models (ClipAssistNet) significantly improved performance over any single model, highlighting the benefit of ensemble methods for complex visual tasks.
  • Real-time Deployment: The model is designed to run efficiently on embedded hardware (NVIDIA Jetson AGX Xavier), achieving 16 FPS, which is crucial for real-time intra-operative feedback.

Key Results

ClipAssistNet achieved a high AUROC of 0.9107 and a specificity of 66.15% at 95% sensitivity on the unseen test set. When evaluated on a per-video basis, the median specificity rose to 81.20% at 95% sensitivity, indicating strong performance across individual surgical procedures. The model's ability to focus on relevant image regions was confirmed by saliency maps.

Limitations and Future Work

The authors acknowledge that despite efforts to create objective annotation guidelines, some subjectivity remains, particularly in cases of poor contrast, leading to labeling noise. This noise, while mitigated by triple annotation on the test set, could still impact model performance. The paper also notes that determining the optimal threshold for triggering the feedback in a real surgical setting requires further usability studies. Future work could explore integrating this feedback into surgical workflows, potentially by automatically detecting surgical phases to trigger the alerts, and further refining the annotation process to minimize subjectivity.

Summary updated

Abstract

Cholecystectomy is one of the most common laparoscopic procedures. A critical phase of laparoscopic cholecystectomy consists in clipping the cystic duct and artery before cutting them. Surgeons can improve the clipping safety by ensuring full visibility of the clipper, while enclosing the artery or the duct with the clip applier jaws. This can prevent unintentional interaction with neighboring tissues or clip misplacement. In this article, we present a novel real-time feedback to ensure safe visibility of the instrument during this critical phase. This feedback incites surgeons to keep the tip of their clip applier visible while operating.
BibTeX
@article{aspart2022clipassistnet,
  author = {Florian Aspart and Jon Lindström Bolmgren and Joël L. Lavanchy and Guido Beldi and Michael S. Woods and Nicolas Padoy and Enes Hosgor},
  title = {{ClipAssistNet: bringing real-time safety feedback to operating rooms}},
  journal = {Int. J. Comput. Assist. Radiol. Surg.},
  year = {2022},
  url = {https://openreview.net/forum?id=MPcEJK9whd},
}
Figure