Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures

Sanat Ramesh, Diego Dall'Alba, Cristians Gonzalez, Tong Yu, Pietro Mascagni, Didier Mutter, Jacques Marescaux, Paolo Fiorini, Nicolas Padoy

Abstract

Automatic segmentation and classification of surgical activity is crucial for providing advanced support in computer-assisted interventions and autonomous functionalities in robot-assisted surgeries. Prior works have focused on recognizing either coarse activities, such as phases, or fine-grained activities, such as gestures. This work aims at jointly recognizing two complementary levels of granularity directly from videos, namely phases and steps. We introduce two correlated surgical activities, phases and steps, for the laparoscopic gastric bypass procedure. We propose a Multi-task Multi-Stage Temporal Convolutional Network (MTMS-TCN) along with a multi-task Convolutional Neural Network (CNN) training setup to jointly predict the phases and steps and benefit from their complementarity to better evaluate the execution of the procedure. We evaluate the proposed method on a large video dataset consisting of 40 surgical procedures (Bypass40). The proposed MTMS-TCN method outperforms in both phase and step recognition by 1-2% in accuracy, precision and recall, compared to single-task methods. Furthermore, for step recognition, MTMS-TCN achieves a superior performance of 3-6% compared to LSTM based models.

Introduction

In the modern Operating Room (OR), computer-assisted systems aim to provide real-time support, automate documentation, and improve patient safety. To achieve this, AI must "understand" the surgical workflow—knowing exactly what is happening at any given moment. Most prior research has focused on recognizing surgical activity at a single level of granularity: either very coarse phases (e.g., "preparation") or very fine gestures (e.g., "suturing"). However, surgery is naturally hierarchical. A phase consists of several steps, and a step consists of several actions. This paper addresses the gap in joint recognition, specifically targeting the complex Laparoscopic Roux-En-Y Gastric Bypass (LRYGB) procedure, which features a highly variable and intricate workflow.

Methods

The authors propose a two-stage, multi-task pipeline designed for online recognition, predicting the current state using only past and present frames.

Feature Extraction (MT-ResNet)

A ResNet-50 backbone extracts visual features from video frames. This backbone is trained in a multi-task setup to predict both the phase and the step simultaneously, helping the model learn features relevant to both levels of granularity.

Temporal Modeling (MTMS-TCN)

The core innovation is the Multi-Task Multi-Stage Temporal Convolutional Network.

  • Multi-Stage: The network uses multiple stages to iteratively refine its predictions.
  • Causal Dilated Convolutions: To enable online processing, it uses causal convolutions (no looking into the future). Dilated convolutions allow the model to have a massive "receptive field," looking back far into the surgery's history to understand the current context without the vanishing gradient issues of RNNs.
  • Multi-Task Heads: At the end of each stage, the model has two "heads" to output predictions for both phases and steps.

The Bypass40 Dataset

The study introduces Bypass40, a significant new benchmark for the surgical AI community. It consists of 40 full-length endoscopic videos of gastric bypass procedures, with an average duration of 110 minutes per video. These are hierarchically labeled with 11 phases and 44 steps. Unlike the common Cholec80 dataset, Bypass40 provides a much deeper look into the mid-level "steps" of a complex procedure.

Examples of hierarchical annotations in the Bypass40 dataset, showing phases (P) and steps (S).

Key Results and Insights

The MTMS-TCN was benchmarked against several state-of-the-art models, including TeCNO and ResNetLSTM.

  • Phase Recognition: Achieved 91.16% accuracy, outperforming single-task models by 1-2%.
  • Step Recognition: Achieved 76.09% accuracy, a 3-6% improvement over LSTM-based models.
  • Joint Accuracy: The "Phase-Step Accuracy" (where both must be correct) reached 75.14%.
  • Hierarchical Synergy: Training a model to recognize both phases and steps at once improves performance for each individually. The "step" prediction provides a fine-grained hint to the "phase" predictor, and vice versa.
  • TCN Superiority: For long surgical videos, TCNs significantly outperform LSTMs, as dilated convolutions provide a more stable temporal window over hour-long durations.

Conclusion

This work presents a multi-task multi-stage temporal convolutional network for surgical activity recognition, demonstrating that joint modeling of phases and steps is beneficial. While class imbalance remains a challenge due to the rarity of certain surgical steps, the proposed method sets a new standard for hierarchical activity recognition in complex laparoscopic procedures. Future work may involve incorporating instrument tracking data to further boost performance and testing generalization on multi-center data.

Summary updated

Abstract

Purpose: Automatic segmentation and classification of surgical activity is crucial for providing advanced support in computer-assisted interventions and autonomous functionalities in robot-assisted surgeries. Prior works have focused on recognizing either coarse activities, such as phases, or fine-grained activities, such as gestures. This work aims at jointly recognizing two complementary levels of granularity directly from videos, namely phases and steps. Method: We introduce two correlated surgical activities, phases and steps, for the laparoscopic gastric bypass procedure. We propose a Multi-task Multi-Stage Temporal Convolutional Network (MTMS-TCN) along with a multi-task Convolutional Neural Network (CNN) training setup to jointly predict the phases and steps and benefit from their complementarity to better evaluate the execution of the procedure. We evaluate the proposed method on a large video dataset consisting of 40 surgical procedures (Bypass40). Results: We present experimental results from several baseline models for both phase and step recognition on the Bypass40 dataset. The proposed MTMS-TCN method outperforms in both phase and step recognition by 1-2% in accuracy, precision and recall, compared to single-task methods. Furthermore, for step recognition, MTMS-TCN achieves a superior performance of 3-6% compared to LSTM based models in accuracy, precision, and recall. Conclusion: In this work, we present a multi-task multi-stage temporal convolutional network for surgical activity recognition, which shows improved results compared to single-task models on the Bypass40 gastric bypass dataset with multi-level annotations. The proposed method shows that the joint modeling of phases and steps is beneficial to improve the overall recognition of each type of activity.
BibTeX
@article{ramesh2021multi,
  author = {Sanat Ramesh and Diego Dall'Alba and Cristians Gonzalez and Tong Yu and Pietro Mascagni and Didier Mutter and Jacques Marescaux and Paolo Fiorini and Nicolas Padoy},
  title = {{Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures}},
  journal = {Int. J. Comput. Assist. Radiol. Surg.},
  year = {2021},
  url = {https://openreview.net/forum?id=53pdVNUCGH},
}
Figure