Abstract
Automatic recognition of fine-grained surgical activities, known as steps, is a critical component for developing intelligent intra-operative computer assistance. However, current vision-based methods rely heavily on large volumes of manually annotated data, which is time-consuming to produce and requires specialized medical expertise. This paper proposes a weakly supervised framework that leverages coarser, easier-to-annotate "phase" labels to improve step recognition. By introducing a novel step-phase dependency loss ($L_{dep}$) and utilizing a Single-Stage Temporal Convolutional Network (SS-TCN) with a ResNet-50 backbone, the method achieves significant performance gains in data-scarce scenarios. The approach is validated on the Bypass40 dataset and the CATARACTS benchmark, demonstrating its effectiveness in reducing the "annotation tax" for surgical AI.
Introduction
In the field of Computer-Assisted Interventions (CAI), real-time understanding of surgical workflows is essential for providing context-aware assistance. This understanding requires the recognition of fine-grained steps—short, visually similar actions that constitute a procedure. While deep learning models excel at this task, they typically require thousands of meticulously labeled video frames.
The primary bottleneck is the "annotation tax": labeling 44 distinct steps across a complex procedure like a Laparoscopic Gastric Bypass requires expert surgeons to spend hours reviewing footage. In contrast, phases (e.g., "Preparation" or "Pouch Creation") are much easier and faster to identify. This research addresses the gap between coarse and fine-grained labeling, exploring how models can learn complex step recognition using only a handful of fully annotated videos supplemented by weak phase-level supervision.
Methodology
The proposed framework employs a spatio-temporal architecture designed for end-to-end learning from weakly annotated videos.
Architecture
- Spatial Feature Extraction: A ResNet-50 backbone processes individual video frames to capture visual features.
- Temporal Modeling: A Single-Stage Temporal Convolutional Network (SS-TCN) analyzes sequences of these features to capture the temporal flow and dependencies of the surgical procedure.
Step-Phase Dependency Loss ($L_{dep}$)
The core innovation is the introduction of a hierarchical constraint. The authors define a binary mapping matrix that encodes the relationship between steps and phases (identifying which steps are valid within a specific phase).
During training, when a video lacks step-level labels but provides phase labels, the model still predicts step probabilities. These predictions are mathematically mapped into a "phase space" using the dependency matrix. The model is then penalized if its predicted phase—derived from its step guesses—deviates from the ground-truth phase label. This forces the model to respect the underlying surgical ontology, effectively narrowing the search space and guiding the model toward correct step identification even without frame-by-frame step labels.
Experimental Setup and Datasets
The method was evaluated on two distinct surgical procedures to ensure generalizability:
- Bypass40: Consists of 40 videos of Laparoscopic Roux-en-Y Gastric Bypass (LRYGB), a complex procedure featuring 11 phases and 44 steps.
- CATARACTS: A public benchmark of 50 cataract surgeries. The authors extended this dataset by adding phase annotations to the existing 19 step labels, creating a hierarchical version for the research community.
Key Results
The dependency loss model (DEP) consistently outperformed fully supervised baselines (FSA) and existing semi-supervised state-of-the-art (SOTA) methods, particularly when labeled data was scarce.
- Efficiency in Low-Data Regimes: On the Bypass40 dataset, using only three fully labeled videos combined with weak phase labels, the model improved accuracy from 45% to 57%—a 12% absolute improvement over the baseline.
- Comparison to SOTA: The proposed method outperformed previous semi-supervised approaches by 10-15% in precision and recall in scenarios where step labels were limited.
- Robustness: The performance boost was consistent across both the long, complex gastric bypass procedures and the shorter, more standardized cataract surgeries.
Discussion and Limitations
While the framework successfully leverages hierarchy as a constraint, some challenges remain. The model occasionally struggles with intra-phase ambiguity, where different steps within the same phase look visually similar (e.g., different types of stapling). Because these steps share a phase label, the dependency loss provides limited guidance for differentiation. Additionally, while the model learns which steps belong to a phase, the current loss function does not explicitly enforce the sequential order of steps within that phase.
Conclusion
This work presents a practical solution to the data annotation bottleneck in surgical activity recognition. By utilizing the inherent hierarchical structure of surgical procedures, the proposed weakly supervised TCN allows for high-performance activity recognition with a fraction of the expert labor. Future work will focus on improving inter-class separation for visually similar steps within phases, potentially through self-supervised contrastive learning techniques.
Summary updated