Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition

Sanat Ramesh, Diego Dall'Alba, Cristians Gonzalez, Tong Yu, Pietro Mascagni, Didier Mutter, Jacques Marescaux, Paolo Fiorini, Nicolas Padoy

Abstract

Automatic recognition of fine-grained surgical activities, known as steps, is a critical component for developing intelligent intra-operative computer assistance. However, current vision-based methods rely heavily on large volumes of manually annotated data, which is time-consuming to produce and requires specialized medical expertise. This paper proposes a weakly supervised framework that leverages coarser, easier-to-annotate "phase" labels to improve step recognition. By introducing a novel step-phase dependency loss ($L_{dep}$) and utilizing a Single-Stage Temporal Convolutional Network (SS-TCN) with a ResNet-50 backbone, the method achieves significant performance gains in data-scarce scenarios. The approach is validated on the Bypass40 dataset and the CATARACTS benchmark, demonstrating its effectiveness in reducing the "annotation tax" for surgical AI.

Introduction

In the field of Computer-Assisted Interventions (CAI), real-time understanding of surgical workflows is essential for providing context-aware assistance. This understanding requires the recognition of fine-grained steps—short, visually similar actions that constitute a procedure. While deep learning models excel at this task, they typically require thousands of meticulously labeled video frames.

The primary bottleneck is the "annotation tax": labeling 44 distinct steps across a complex procedure like a Laparoscopic Gastric Bypass requires expert surgeons to spend hours reviewing footage. In contrast, phases (e.g., "Preparation" or "Pouch Creation") are much easier and faster to identify. This research addresses the gap between coarse and fine-grained labeling, exploring how models can learn complex step recognition using only a handful of fully annotated videos supplemented by weak phase-level supervision.

Methodology

The proposed framework employs a spatio-temporal architecture designed for end-to-end learning from weakly annotated videos.

Architecture

  1. Spatial Feature Extraction: A ResNet-50 backbone processes individual video frames to capture visual features.
  2. Temporal Modeling: A Single-Stage Temporal Convolutional Network (SS-TCN) analyzes sequences of these features to capture the temporal flow and dependencies of the surgical procedure.

Step-Phase Dependency Loss ($L_{dep}$)

The core innovation is the introduction of a hierarchical constraint. The authors define a binary mapping matrix that encodes the relationship between steps and phases (identifying which steps are valid within a specific phase).

During training, when a video lacks step-level labels but provides phase labels, the model still predicts step probabilities. These predictions are mathematically mapped into a "phase space" using the dependency matrix. The model is then penalized if its predicted phase—derived from its step guesses—deviates from the ground-truth phase label. This forces the model to respect the underlying surgical ontology, effectively narrowing the search space and guiding the model toward correct step identification even without frame-by-frame step labels.

Experimental Setup and Datasets

The method was evaluated on two distinct surgical procedures to ensure generalizability:

  • Bypass40: Consists of 40 videos of Laparoscopic Roux-en-Y Gastric Bypass (LRYGB), a complex procedure featuring 11 phases and 44 steps.
  • CATARACTS: A public benchmark of 50 cataract surgeries. The authors extended this dataset by adding phase annotations to the existing 19 step labels, creating a hierarchical version for the research community.

Key Results

The dependency loss model (DEP) consistently outperformed fully supervised baselines (FSA) and existing semi-supervised state-of-the-art (SOTA) methods, particularly when labeled data was scarce.

  • Efficiency in Low-Data Regimes: On the Bypass40 dataset, using only three fully labeled videos combined with weak phase labels, the model improved accuracy from 45% to 57%—a 12% absolute improvement over the baseline.
  • Comparison to SOTA: The proposed method outperformed previous semi-supervised approaches by 10-15% in precision and recall in scenarios where step labels were limited.
  • Robustness: The performance boost was consistent across both the long, complex gastric bypass procedures and the shorter, more standardized cataract surgeries.

Discussion and Limitations

While the framework successfully leverages hierarchy as a constraint, some challenges remain. The model occasionally struggles with intra-phase ambiguity, where different steps within the same phase look visually similar (e.g., different types of stapling). Because these steps share a phase label, the dependency loss provides limited guidance for differentiation. Additionally, while the model learns which steps belong to a phase, the current loss function does not explicitly enforce the sequential order of steps within that phase.

Conclusion

This work presents a practical solution to the data annotation bottleneck in surgical activity recognition. By utilizing the inherent hierarchical structure of surgical procedures, the proposed weakly supervised TCN allows for high-performance activity recognition with a fraction of the expert labor. Future work will focus on improving inter-class separation for visually similar steps within phases, potentially through self-supervised contrastive learning techniques.

Summary updated

Abstract

Automatic recognition of fine-grained surgical activities, called steps, is a challenging but crucial task for intelligent intra-operative computer assistance. The development of current vision-based activity recognition methods relies heavily on a high volume of manually annotated data. This data is difficult and time-consuming to generate and requires domain-specific knowledge. In this work, we propose to use coarser and easier-to-annotate activity labels, namely phases, as weak supervision to learn step recognition with fewer step annotated videos. We introduce a step-phase dependency loss to exploit the weak supervision signal. We then employ a Single-Stage Temporal Convolutional Network (SS-TCN) with a ResNet-50 backbone, trained in an end-to-end fashion from weakly annotated videos, for temporal activity segmentation and recognition. We extensively evaluate and show the effectiveness of the proposed method on a large video dataset consisting of 40 laparoscopic gastric bypass procedures and the public benchmark CATARACTS containing 50 cataract surgeries.
BibTeX
@article{ramesh2023weakly,
  author = {Sanat Ramesh and Diego Dall'Alba and Cristians Gonzalez and Tong Yu and Pietro Mascagni and Didier Mutter and Jacques Marescaux and Paolo Fiorini and Nicolas Padoy},
  title = {{Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2023},
  url = {https://openreview.net/forum?id=wA7shRsgeM},
}
Figure