Multilevel effective surgical workflow recognition in robotic left lateral sectionectomy with deep learning: experimental research

Yanzhe Liu, Shang Zhao, Gong Zhang, Xiuping Zhang, Minggen Hu, Xuan Zhang, Chenggang Li, S. Kevin Zhou, Rong LiuInternational Journal of Surgery 2023 · DOI

TL;DR

This paper introduces a novel deep learning approach for recognizing surgical workflows at multiple granularities (steps, tasks, and activities) in robotic left lateral sectionectomy. By creating a detailed annotated dataset and developing a hybrid CNN-GNN model, they demonstrate significantly improved recognition accuracy when "under-effective" (non-essential) frames are filtered out, paving the way for more autonomous robotic surgery.

What Problem Does This Paper Address?

Automating surgical workflow recognition is crucial for developing intelligent surgical systems, including those for autonomous robotic surgery. This involves understanding the sequence of actions performed during an operation. However, current methods often focus on broad surgical phases, lacking the fine-grained detail needed for true autonomy. A major challenge is the presence of "under-effective" frames in surgical videos – moments where instruments are present but not actively engaged with tissue, or where the view is obscured. These frames can degrade the performance of recognition models and make it harder for surgeons to learn from videos. This paper aims to address this by developing a system that can recognize surgical workflows at multiple levels of detail and demonstrating the benefit of filtering out these under-effective frames.

Methods

The core of this research lies in a hybrid deep learning model that combines Convolutional Neural Networks (CNNs) and Graph Neural Networks (GNNs).

  • CNN Encoder: This module extracts low-level structural and contextual features from individual endoscopic images. It uses a ResNet50 architecture, pre-trained on ImageNet, to capture spatial information.
  • GNN Module: This module refines the high-level context features extracted by the CNN. It uses a graph convolutional operator to aggregate frame features based on their cosine similarity, creating an "affinity matrix" that captures relationships between frames. This allows the model to understand how features evolve over time and across different frames.
  • Fusion Module: The features from both the CNN and GNN are concatenated to provide a comprehensive representation for surgical behavior identification.

The paper also introduces a novel concept of "effective" vs. "under-effective" frames. Effective frames are those where surgical instruments are actively interacting with tissue, contributing directly to the surgical objective. Under-effective frames include instrument-free states, non-visualized activities (idle time), preparation states before activation, and activities of auxiliary instruments. The model is trained to recognize surgical steps, tasks, and activities, and its performance is evaluated both with and without the filtering of under-effective frames.

Datasets

The authors created a new dataset specifically for this research:

  • Robotic Left Lateral Sectionectomy (RLLS) Dataset: This dataset comprises video data from 45 robotic left lateral sectionectomy procedures performed between December 2016 and May 2019.
  • It contains a total of 4,383,516 annotated frames.
  • Of these, 2,418,468 frames are classified as "effective."
    • The annotations are hierarchical, covering three levels: 4 Steps, 12 Tasks, and 26 Activities.

Key Insights and Ideas

  • The Importance of "Effective" Frames: The most significant insight is that filtering out "under-effective" frames dramatically improves surgical workflow recognition accuracy. This suggests that focusing on the core actions, rather than including all visual information, leads to more robust and efficient models.
  • Multilevel Granularity is Key: Recognizing surgical workflows at multiple levels (Step, Task, Activity) provides a more comprehensive understanding than just identifying broad phases. This finer-grained recognition is essential for developing more sophisticated autonomous surgical systems.
  • Hybrid CNN-GNN Power: Combining CNNs for spatial feature extraction with GNNs for temporal and relational context modeling proves to be a powerful approach for complex surgical video analysis.
  • Standardization Benefits AI: The standardized nature of robotic surgery, particularly the da Vinci system, provides a unique advantage for developing AI models due to consistent camera parameters and instrument data, facilitating transfer learning.

Key Results

  • Overall Recognition Accuracy:
    • Steps: 82% accuracy, 81% precision
    • Tasks: 80% accuracy, 76% precision
    • Activities: 79% accuracy, 60% precision
    • Under-effective frames: 85% accuracy, 85% precision
  • Effective Surgical Workflow Recognition (after filtering under-effective frames):
    • Steps: 96% accuracy, 95% precision
    • Tasks: 88% accuracy, 80% precision
    • Activities: 82% accuracy, 68% precision
  • Precision Improvements: Filtering under-effective frames led to significant precision increases, particularly for Steps (from 81% to 95%) and Activities (from 60% to 68%).
  • Dataset Size: The dataset is substantial, with over 4.3 million annotated frames, providing a strong foundation for training deep learning models.

Limitations and Future Work

  • Single-Center Data: The dataset was acquired from a single center, which might lead to overfitting and limit generalizability to other institutions or surgical variations.
  • Inadequate Recognition for Some Events: The recognition rate for certain tasks and activities remains insufficient for practical application, often due to limited training data for those specific events (e.g., Hemostasis of perihepatic tissues).
  • Subtle Visual Differences: Distinguishing between visually similar events (e.g., different types of stapler use) can be challenging and requires more data.
  • Lack of Kinematic Data: The study did not incorporate kinematic data from surgical instruments, which could further enhance motion modeling and gesture segmentation.
  • Future Directions:
    • Increase training sample size and diversity to improve model accuracy and robustness.
  • Incorporate kinematic data from robotic systems for deeper motion analysis.
  • Explore transfer learning to adapt models to other surgical procedures.
  • Develop more sophisticated ethical frameworks for AI in surgery, addressing data privacy, security, and algorithmic bias.

Summary updated

Abstract

In this study, the authors created a dataset of 45 RLLS cases with multilevel annotations and developed a hybrid deep learning model for surgical workflow recognition. The authors demonstrated a fairly higher accuracy in multilevel effective surgical workflow recognition when under-effective frames were removed. Our research could be helpful in the development of autonomous robotic surgery.
BibTeX
@article{liu2023multilevel,
  author = {Yanzhe Liu and Shang Zhao and Gong Zhang and Xiuping Zhang and Minggen Hu and Xuan Zhang and Chenggang Li and S. Kevin Zhou and Rong Liu},
  title = {{Multilevel effective surgical workflow recognition in robotic left lateral sectionectomy with deep learning: experimental research}},
  journal = {International Journal of Surgery},
  year = {2023},
  doi = {10.1097/js9.0000000000000559},
}
Figure