Prediction of laparoscopic procedure duration using unlabeled, multimodal sensor data

Sebastian Bodenstedt, Martin Wagner, Lars Mündermann, Hannes Kenngott, Beat P. Müller-Stich, Michael Breucha, Sören Torge Mees, Jürgen Weitz, Stefanie Speidel

Abstract

Efficient operating room (OR) scheduling is critical for hospital logistics, yet traditionally relies on subjective surgeon estimates or labor-intensive manual phase-tracking. This paper proposes an automated approach to predict remaining surgery duration by fusing endoscopic video with data from surgical devices. By employing self-supervised learning on unlabeled temporal sequences, the developed multimodal model achieves a 37% average error across diverse procedure types, significantly outperforming traditional baselines without requiring manual annotation.

Introduction

The core challenge in OR management is unpredictability. Surgical workflows vary significantly based on patient anatomy and unforeseen complications. While previous AI attempts at duration prediction have shown promise, they typically rely on "surgical phase detection," which requires thousands of hours of manually labeled video data. This creates a significant labeling bottleneck and limits the ability of models to generalize across different types of procedures.

To address these gaps, the authors developed a context-aware method that utilizes multimodal data already generated during laparoscopic procedures—such as video feeds and device telemetry—to predict progress without explicit human labeling.

Methods

The researchers proposed a suite of models based on Recurrent Convolutional Neural Networks (CNNs). The architecture utilizes a CNN to extract features from each frame and a Gated Recurrent Unit (GRU) to maintain a temporal history of the surgery.

The VTD-Net architecture fuses image features, device data, tool presence, and procedure type into a GRU for duration prediction.

Key components of the multimodal fusion include:

  • Self-Supervised Vision (V-Net): To avoid manual labels, the vision model was pretrained using a Siamese approach. The network learned the temporal flow of surgery by determining the correct order of random frames, effectively learning what "progress" looks like.
  • Tool Presence (T-Net): A ResNet-152 detects the presence of 12 surgical tools, serving as a proxy for specific surgical tasks.
  • Device Data (D-Net): The model integrates 14 discrete signals from OR equipment, including insufflator gas flow/pressure, light intensity, and camera exposure settings.
  • Multimodal Fusion (VTD-Net): The final model concatenates these streams into a single input for the GRU.

Results and Discussion

The study evaluated these models on two datasets: MultiType (80 diverse interventions) and Cholec80 (80 standardized gallbladder removals). The results demonstrate that while vision is the primary driver of accuracy, multimodal fusion provides critical context.

Relative error (%) of various model configurations across surgery quartiles (Q1-Q4) and mean performance.

As shown in the results table, the full VTD-Net achieved an overall average error of 37% on the MultiType dataset. Accuracy improved as the surgery progressed; by the halftime mark, the error dropped to approximately 28%. In contrast, a "Naive" baseline of guessing the average duration for a surgery type resulted in a 57% error, highlighting the significant predictive value of the AI model.

On the standardized Cholec80 benchmark, the vision-tool model (VT-Net) achieved a 23% average error, outperforming previous state-of-the-art methods like Twinanda et al.’s RSDNet (25% error).

Conclusion

This research demonstrates that self-supervised, multimodal learning can effectively predict surgical duration without the need for expensive manual annotations. While the model performs best on standardized procedures, it shows robust improvements over traditional baselines even in diverse surgical environments. Future work may involve incorporating additional data streams, such as anesthesia telemetry, to further refine predictions and account for intraoperative complications.

Summary updated

Abstract

Purpose The course of surgical procedures is often unpredictable, making it difficult to estimate the duration of procedures beforehand. This uncertainty makes scheduling surgical procedures a difficult task. A context-aware method that analyses the workflow of an intervention online and automatically predicts the remaining duration would alleviate these problems. As basis for such an estimate, information regarding the current state of the intervention is a requirement. Methods Today, the operating room contains a diverse range of sensors. During laparoscopic interventions, the endoscopic video stream is an ideal source of such information. Extracting quantitative information from the video is challenging though, due to its high dimensionality. Other surgical devices (e.g., insufflator, lights, etc.) provide data streams which are, in contrast to the video stream, more compact and easier to quantify. Though whether such streams offer sufficient information for estimating the duration of surgery is uncertain. In this paper, we propose and compare methods, based on convolutional neural networks, for continuously predicting the duration of laparoscopic interventions based on unlabeled data, such as from endoscopic image and surgical device streams. Results The methods are evaluated on 80 recorded laparoscopic interventions of various types, for which surgical device data and the endoscopic video streams are available. Here the combined method performs best with an overall average error of 37% and an average halftime error of approximately 28%. Conclusion In this paper, we present, to our knowledge, the first approach for online procedure duration prediction using unlabeled endoscopic video data and surgical device data in a laparoscopic setting. Furthermore, we show that a method incorporating both vision and device data performs better than methods based only on vision, while methods only based on tool usage and surgical device data perform poorly, showing the importance of the visual channel.
BibTeX
@article{bodenstedt2019prediction,
  author = {Sebastian Bodenstedt and Martin Wagner and Lars Mündermann and Hannes Kenngott and Beat P. Müller-Stich and Michael Breucha and Sören Torge Mees and Jürgen Weitz and Stefanie Speidel},
  title = {{Prediction of laparoscopic procedure duration using unlabeled, multimodal sensor data}},
  journal = {Int. J. Comput. Assist. Radiol. Surg},
  year = {2019},
  doi = {10.1007/s11548-019-01966-6},
}
Figure