Abstract
Accurate surgery duration estimation is necessary for optimal OR planning, which plays an important role in patient comfort and safety as well as resource optimization. It is, however, challenging to preoperatively predict surgery duration since it varies significantly depending on the patient condition, surgeon skills, and intraoperative situation. In this paper, we propose a deep learning pipeline, referred to as RSDNet, which automatically estimates the remaining surgery duration (RSD) intraoperatively by using only visual information from laparoscopic videos. Previous state-of-the-art approaches for RSD prediction are dependent on manual annotation, whose generation requires expensive expert knowledge and is time-consuming, especially considering the numerous types of surgeries performed in a hospital and the large number of laparoscopic videos available. A crucial feature of RSDNet is that it does not depend on any manual annotation during training, making it easily scalable to many kinds of surgeries. The generalizability of our approach is demonstrated by testing the pipeline on two large datasets containing different types of surgeries: 120 cholecystectomy and 170 gastric bypass videos. The experimental results also show that the proposed network significantly outperforms a traditional method of estimating RSD without utilizing manual annotation. Further, this work provides a deeper insight into the deep learning network through visualization and interpretation of the features that are automatically learned.
Introduction
Efficient Operating Room (OR) management is critical for hospital logistics, patient safety, and resource optimization. Accurate estimates of Remaining Surgery Duration (RSD) facilitate better scheduling, anesthesia management, and staff transitions. However, manual estimation by clinical staff is often inaccurate, with studies showing that surgeons and anesthesiologists frequently underestimate surgery times by 30% to 160%.
While AI has shown promise in surgical phase recognition, most existing models rely on supervised learning, requiring hundreds of hours of manual labeling by experts. This "annotation bottleneck" limits the scalability of AI tools across the vast variety of surgical procedures. RSDNet addresses this by utilizing a self-supervised training strategy that eliminates the need for manual annotations by automatically generating "progress" labels from raw video data.
Methods
The RSDNet pipeline employs a two-stage deep learning architecture designed to process laparoscopic video feeds in real-time.
- Feature Extraction (CNN): A ResNet-152 convolutional neural network processes individual video frames. Instead of recognizing specific phases, it is trained on a "Progress Estimation" task. Since every video has a known start and end, progress labels (0% to 100%) can be generated automatically post-surgery for training.
- Temporal Modeling (LSTM): To capture the sequential nature of surgery, visual features from the CNN are fed into a Long Short-Term Memory (LSTM) network.
- Multi-Task Learning: The LSTM simultaneously predicts the current progress percentage and regresses the actual remaining time in minutes. It also incorporates "elapsed time" ($t_{el}$) as an input feature to leverage the mathematical link between progress, elapsed time, and remaining time.
The training is conducted in two steps: first, fine-tuning the CNN on progress labels, followed by training the LSTM on feature sequences while keeping the CNN weights frozen.
Datasets and Experimental Setup
The model was evaluated on two distinct datasets:
- Cholec120: 120 cholecystectomy videos (avg. duration 38 mins), characterized by high variance in length.
- Bypass170: 170 gastric bypass videos (avg. duration 115 mins), representing longer and more complex procedures.
Results
RSDNet demonstrated robust performance across both datasets, significantly outperforming "Naive" historical average baselines, particularly for outlier "long" surgeries where historical averages fail.
- Cholec120: Achieved a Mean Absolute Error (MAE) of 8.1 minutes, comparable to state-of-the-art models that require manual phase annotations.
- Bypass170: Achieved an MAE of 15.6 minutes on these significantly longer procedures.
Key insights from the results include:
- Progress as a Proxy: Continuous progress estimation is a powerful signal that allows the CNN to learn semantically meaningful features (e.g., identifying early vs. late stages) without explicit phase labels.
- Direct Regression Superiority: Direct regression of RSD in minutes proved more accurate than calculating it indirectly from progress predictions.
- Internal Representation: Visualization of LSTM activations revealed "clock cells" that track surgical progress linearly and cells that respond to specific visual cues, such as the appearance of a specimen bag.
Conclusion and Future Work
RSDNet provides a scalable, self-supervised solution for intraoperative RSD prediction. By removing the need for manual annotations, it can be easily adapted to new surgical procedures using existing video archives. Future work includes exploring end-to-end training of the CNN-LSTM pipeline and integrating non-visual data (e.g., patient vitals, surgeon experience) to further enhance prediction accuracy and handle unexpected intraoperative delays.
Summary updated