Introduction
Surgical Phase Recognition (SPR) is a cornerstone of surgical data science, enabling computers to understand the specific steps of a procedure in real-time. While previous research has focused on linear surgeries like gallbladder removals, this study benchmarks Artificial Intelligence against urologists in the context of Robot-Assisted Partial Nephrectomy (RAPN)—a complex, non-linear procedure. The study investigates the impact of temporal context on performance and evaluates the gap between human expertise and state-of-the-art neural networks.
Methods and Participants
The researchers conducted a comparative study involving a human cohort and an AI pipeline. The human survey recruited 100 participants across three levels of expertise: medical students, surgical trainees, and consultants.
Participants were tasked with identifying surgical phases from 17 random moments, presented first as single still frames and then as 10-second video snippets. Concurrently, three AI architectures were evaluated: a frame-based ResNet50, a temporal ResNet50-LSTM, and TeCNO, a multi-stage model designed for full workflow analysis.
Datasets
The study utilized two primary datasets:
- Cholec80: A standard benchmark of 80 laparoscopic cholecystectomy videos used for model validation.
- RAPN Dataset: A challenging dataset of 143 robotic partial nephrectomy recordings, featuring 15 distinct phases and a high degree of procedural non-linearity.
Results: The "Motion Premium"
A key finding was the significant performance boost provided by temporal context. For all human groups, moving from still frames to 10-second video snippets improved accuracy. Consultants, for instance, saw a 12% increase in performance when given motion cues.
Despite human expertise, the AI models demonstrated superior raw classification capabilities. The TeCNO model achieved an accuracy of 61.6%, notably higher than the 53.2% average achieved by consultants on video snippets. This suggests that while human expertise is grounded in anatomical knowledge, AI can leverage large-scale data to master complex visual patterns.
Key Insights
- Shared Ambiguities: Both humans and AI struggled with specific phases like "Hilar Control," which lack distinct visual landmarks.
- Data vs. Knowledge: AI performance dropped to student levels when trained on only 25% of the data, highlighting its reliance on volume compared to human anatomical reasoning.
- Confidence Disconnect: While consultants expressed higher confidence than students, their subjective certainty did not always correlate with higher objective accuracy.
Conclusion and Future Work
The study concludes that while AI currently leads in raw accuracy for SPR in complex surgeries, both humans and machines benefit immensely from temporal context. Future research will explore longer context windows (minutes rather than seconds) and multi-task learning to better integrate tool and organ recognition into phase classification. Furthermore, the discrepancy between expert confidence and accuracy suggests a need for caution regarding confirmation bias in manual data annotation.
Summary updated