Abstract
Robust and automated surgical workflow detection in real time is a core component of the future intelligent operating room. Based on this technology, it can help medical staff to automate and intelligently complete many routine activities during surgery. Recognition of surgical workflow based on traditional pattern recognition methods requires a large number of labeled surgical video data. However, the labeled surgical video data requires expert knowledge and it is difficult and time consuming to collect a sufficient number of labeled surgical video data in the medical field. Therefore, this paper proposes a semi-supervised spatio-temporal convolutional network for the recognition of surgical workflow based on convolutional neural networks and temporal-recursive networks. Firstly, we build a spatial convolutional extraction feature network based on unsupervised generative adversarial learning. Then, we build a bridge between low-level surgical video features and high-level surgical workflow semantics based on an unsupervised temporal-ordered network learning approach. Finally, we use the semi-supervised learning method to integrate the spatial model and the temporal model to fine-tune the network, and realize the intelligent recognition of the surgical workflow at a low cost to efficiently determine the progress of the surgical workflow. We performed some experiments for validating the mode based on m2cai16-workflow dataset. It shows that the proposed model can effectively extract the surgical feature and determine the surgical workflow. The Jaccard score of the model reaches 71.3%, and the accuracy of the model reaches 85.8%.
Introduction
The concept of the "intelligent operating room" hinges on context-aware systems capable of monitoring surgical progress in real-time. Such technology is essential for the automatic indexing of surgical videos, estimating remaining operative time, and providing early warnings for critical intraoperative events. However, a significant barrier to deploying deep learning models in this domain is the "data scarcity" problem. Training robust models typically requires massive amounts of expert-labeled video data, which is both expensive and time-consuming to acquire.
This research addresses this challenge by proposing a semi-supervised framework that leverages unlabeled surgical video to learn meaningful representations, thereby reducing the dependency on large-scale manual annotation.
Methodology
The proposed "spatio-tempora-Net" utilizes a three-stage learning process to capture both the visual appearance of surgical scenes and the logical progression of surgical steps.
1. Unsupervised Spatial Learning
The model first learns to understand the visual environment of the operating room using a Generative Adversarial Network (GAN). A "Generator" creates synthetic surgical images while a "Discriminator" learns to distinguish them from real frames. Through this adversarial process, the Discriminator becomes a highly effective feature extractor, capturing the "vocabulary" of surgical tools and anatomy without requiring explicit labels.
2. Unsupervised Temporal Learning
To capture the sequential nature of surgery, the authors employ a Long Short-Term Memory (LSTM) network. The model is trained on a self-supervised "sorting task": given two frames, it must predict their chronological order. This forces the network to learn the inherent logic of surgical procedures—for instance, recognizing that trocar placement must precede organ dissection.
3. Semi-supervised Fine-tuning
In the final stage, the spatial and temporal components are integrated. A small set of labeled data is used to fine-tune the entire stack, mapping the learned features to specific surgical phases.
Experimental Evaluation
The model was validated using the m2cai16-workflow dataset, which contains 41 videos of laparoscopic cholecystectomy (gallbladder removal) procedures. The dataset is categorized into eight distinct phases, such as Trocar placement and gallbladder retraction.
The experimental setup involved 27 training videos and 14 testing videos. By using unsupervised pre-training, the model achieved faster convergence and higher final performance compared to standard supervised baselines.
Results and Discussion
The semi-supervised spatio-temporal model demonstrated significant improvements over traditional supervised CNN-LSTM architectures:
- Accuracy: The model achieved an overall accuracy of 85.8%, a substantial increase over the 72.2% seen in baseline supervised models.
- Jaccard Score: A score of 71.3% was reached, indicating high precision in identifying the transitions between surgical phases.
The success of the GAN-based feature extraction was further evidenced by the quality of generated surgical images, proving the model's deep understanding of the visual domain.
Conclusion
This work demonstrates that semi-supervised learning is a viable path toward robust surgical workflow recognition, even with limited labeled data. While the model performs well, future work will focus on addressing data imbalances in shorter surgical phases and moving toward "fine-grained" action detection, such as specific suturing or cutting movements, to provide more granular support in the operating room.
Summary updated