Abstract
Automatic recognition of fine-grained surgical activities, called steps, is a challenging but crucial task for intelligent intra-operative computer assistance. The development of current vision-based activity recognition methods relies heavily on a high volume of manually annotated data. This data is difficult and time-consuming to generate and requires domain-specific knowledge. In this work, we propose to use coarser and easier-to-annotate activity labels, namely phases, as weak supervision to learn step recognition with fewer step annotated videos. We introduce a step-phase dependency loss to exploit the weak supervision signal. We then employ a Single-Stage Temporal Convolutional Network (SS-TCN) with a ResNet-50 backbone, trained in an end-to-end fashion from weakly annotated videos, for temporal activity segmentation and recognition. We extensively evaluate and show the effectiveness of the proposed method on a large video dataset consisting of 40 laparoscopic gastric bypass procedures and the public benchmark CATARACTS containing 50 cataract surgeries.
Introduction
Intelligent intra-operative assistance requires AI systems to understand surgical workflows at a granular level. This understanding is typically categorized into broad phases (e.g., preparation, cavity exploration) and specific, fine-grained steps (e.g., horizontal stapling, mesenteric opening). While phase recognition has seen significant progress, recognizing fine-grained steps remains a bottleneck due to the "annotation gap."
Annotating surgical steps is labor-intensive and requires expert medical knowledge. Furthermore, many steps exhibit low inter-class variance, making them visually similar and difficult for standard models to distinguish. This research addresses the scarcity of expert-labeled step data by leveraging "weak supervision" from coarser, more easily obtained phase labels.
Methodology
The proposed framework utilizes an end-to-end spatio-temporal model designed for potential real-time application.
Architecture
- Visual Feature Extraction: A ResNet-50 backbone is employed to extract features from individual video frames.
- Temporal Modeling: A Single-Stage Temporal Convolutional Network (SS-TCN) processes these features. By using causal convolutions, the model maintains temporal consistency by looking only at current and past frames, a prerequisite for online surgical assistance.
Step-Phase Dependency Loss ($L_{dep}$)
The core contribution is a novel loss function that exploits the hierarchical relationship between steps and phases. A mapping matrix ($M_{s \to p}$) encodes the medical logic: it defines which steps are valid within a specific phase.
- In fully supervised scenarios (videos with step labels), standard cross-entropy loss is used.
- In weakly supervised scenarios (videos with only phase labels), the model predicts steps, which are then mapped into "phase space" via the matrix. The loss is calculated by comparing these mapped predictions against the ground-truth phase labels.
Experimental Setup
The method was evaluated on two distinct surgical datasets, illustrating its versatility across different medical domains.
- Bypass40: Comprises 40 videos of Laparoscopic Roux-en-Y gastric bypass (LRYGB), annotated with 11 phases and 44 steps.
- CATARACTS: A public benchmark of 50 cataract surgeries. The authors extended this dataset with new phase annotations (5 phases, 19 steps) to support the weakly supervised learning task.
Results and Discussion
The "DEP" (Dependency Loss) model consistently outperformed fully supervised baselines and existing semi-supervised teacher-student approaches.
Performance in Low-Data Regimes
The model showed remarkable resilience when step-labeled data was scarce. On the CATARACTS dataset, using only three step-annotated videos (12% of the training set) alongside weak phase labels, the model achieved 66.78% accuracy. This is a significant jump from the 48.47% achieved by a purely supervised model on the same small subset.
Precision and Recall Improvements
On the Bypass40 dataset, the inclusion of phase-level weak supervision led to a 10-13% improvement in precision and recall when training with only three step-labeled videos. The results indicate that adding "cheap" phase labels provides a more substantial performance boost than simply increasing the volume of "expensive" step labels.
Hierarchical Constraints
The hierarchical mapping acts as a powerful prior, effectively pruning the search space. By knowing the current phase, the model can rule out impossible steps, drastically reducing "out-of-phase" errors. This approach prevents the model from inheriting noise from pseudo-labels, as it trains directly on the ground-truth weak signal.
Conclusion
This work demonstrates that coarse phase labels can effectively bridge the annotation gap for fine-grained surgical step recognition. By training directly on the weak signal through a dependency loss, the model achieves competitive performance with significantly less expert annotation. While the model improves step recognition, distinguishing between visually identical steps within the same phase remains a challenge. Future research will explore advanced visual representations to further refine recognition of these high-similarity actions.
Summary updated