RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations

Andru Putra Twinanda, Gaurav Yengera, Didier Mutter, Jacques Marescaux, Nicolas Padoy

TL;DR

RSDNet is a deep learning pipeline that predicts the remaining duration of a surgery in real-time using only laparoscopic video feeds. Its breakthrough lies in a self-supervised training strategy: it eliminates the need for expensive, manual expert annotations by automatically generating "progress" labels from raw video data, achieving performance comparable to models trained on human-labeled datasets.

What Problem Does This Paper Address?

Efficient Operating Room (OR) management is a logistical nightmare. Accurate estimates of Remaining Surgery Duration (RSD) are critical for scheduling the next patient, managing anesthesia, and optimizing staff transitions. However, humans are notoriously bad at this: studies cited in the paper show that surgeons and anesthesiologists often underestimate surgery times by 30% to 160%.

While AI has shown promise in surgical phase recognition, most models rely on "supervised learning," which requires surgeons to spend hundreds of hours manually labeling video frames (e.g., "this is the suturing phase"). This "annotation bottleneck" makes it nearly impossible to scale AI tools to the thousands of different surgical procedures performed worldwide. RSDNet tackles this by asking: Can we train a model to understand surgical progress using only the video's timeline?

Methods

The researchers proposed a two-stage deep learning architecture:

  1. Feature Extraction (CNN): They used a ResNet-152 (a deep convolutional neural network) to process individual video frames. Instead of training it to recognize specific surgical phases, they trained it on a Progress Estimation task. Since every video has a start (0%) and an end (100%), the "progress" label for any frame can be calculated automatically after the surgery is over.
  2. Temporal Modeling (LSTM): Because surgery is a sequence of events, the model needs "memory." The visual features from the CNN are fed into a Long Short-Term Memory (LSTM) network.
  3. The Multi-Task Twist: The LSTM is designed to perform two tasks simultaneously: predicting the current progress percentage and regressing the actual remaining time in minutes. To improve accuracy, the model also takes the "elapsed time" as an additional input feature, recognizing that the relationship between progress, elapsed time, and remaining time is mathematically linked.

The pipeline is trained in two steps: first, the CNN is fine-tuned on progress labels; then, its weights are frozen, and the LSTM is trained on the resulting feature sequences.

Datasets

The authors evaluated RSDNet on two significant datasets:

  • Cholec120: 120 videos of cholecystectomy (gallbladder removal). These are relatively short (avg. 38 mins) but vary in length by a factor of eight.
  • Bypass170: A larger, more complex dataset of 170 gastric bypass videos. These are much longer (avg. 115 mins) and represent a significant challenge for duration prediction due to high procedural variance.

Key Insights and Ideas

  • Progress as a Proxy for Phase: The most significant insight is that "progress estimation" (a continuous 0 to 1 value) is a powerful enough signal for a CNN to learn semantically meaningful features. The model learns what "early," "middle," and "late" surgery looks like without ever being told what a "clipper" or a "hook" is.
  • Direct vs. Indirect Regression: The authors found that directly regressing the RSD (minutes left) is more accurate than calculating it indirectly from a progress prediction.
  • Interpretable "Clock" Cells: By visualizing the internal activations of the LSTM, the researchers found specific "cells" that acted like internal timers. Some cells' activations increased linearly as the surgery progressed, while others responded specifically to visual cues like the appearance of a "specimen bag," which signals the surgery is nearly over.
  • Scalability: Because the system requires zero manual labeling, it can theoretically be deployed on any surgical archive to "self-train" on new types of procedures.

Key Results

  • Cholec120: RSDNet achieved a Mean Absolute Error (MAE) of 8.1 minutes. This is statistically equivalent to "TimeLSTM," a previous state-of-the-art model that did require manual phase annotations.
  • Bypass170: On these much longer surgeries, the model achieved an MAE of 15.6 minutes.
  • Comparison to Baselines: RSDNet significantly outperformed "Naive" approaches (which simply use historical averages). For "long" surgeries (outliers), the Naive approach failed miserably (35+ min error), while RSDNet remained much more robust by adapting to the visual evidence of the specific case.

Limitations and Future Work

  • Two-Step Training: Currently, the CNN and LSTM are trained separately due to memory constraints of long surgical videos. Training the entire pipeline "end-to-end" might further improve accuracy.
  • Visual Only: The model relies exclusively on the endoscopic camera. It does not yet incorporate other available OR data, such as patient vitals, surgeon experience levels, or preoperative complexity scores, which could provide vital context for unexpected delays.
  • The "Black Box" of Progress: While the model is accurate, it can still be "fooled" by unusual visual artifacts or non-standard surgical techniques that it hasn't seen in its training set.

Summary updated

Abstract

Accurate surgery duration estimation is necessary for optimal OR planning, which plays an important role in patient comfort and safety as well as resource optimization. It is, however, challenging to preoperatively predict surgery duration since it varies significantly depending on the patient condition, surgeon skills, and intraoperative situation. In this paper, we propose a deep learning pipeline, referred to as RSDNet, which automatically estimates the remaining surgery duration (RSD) intraoperatively by using only visual information from laparoscopic videos. The previous state-of-the-art approaches for RSD prediction are dependent on manual annotation, whose generation requires expensive expert knowledge and is time-consuming, especially considering the numerous types of surgeries performed in a hospital and the large number of laparoscopic videos available. A crucial feature of RSDNet is that it does not depend on any manual annotation during training, making it easily scalable to many kinds of surgeries. The generalizability of our approach is demonstrated by testing the pipeline on two large datasets containing different types of surgeries: 120 cholecystectomy and 170 gastric bypass videos. The experimental results also show that the proposed network significantly outperforms a traditional method of estimating RSD without utilizing manual annotation. Further, this paper provides a deeper insight into the deep learning network through visualization and interpretation of the features that are automatically learned.
BibTeX
@article{twinanda2019rsdnet,
  author = {Andru Putra Twinanda and Gaurav Yengera and Didier Mutter and Jacques Marescaux and Nicolas Padoy},
  title = {{RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2019},
  url = {https://openreview.net/forum?id=Wl4xow04ur},
}

Loading related content...

Figure