Surgeons vs. Computer Vision: A comparative analysis on surgical phase recognition capabilities

Marco Mezzina, Pieter De Backer, Tom Vercauteren, Matthew B. Blaschko, Alexandre Mottrie, Tinne Tuytelaars

Introduction

Surgical Phase Recognition (SPR) is a cornerstone of surgical data science, enabling computers to understand the specific steps of a procedure in real-time. While previous research has focused on linear surgeries like gallbladder removals, this study benchmarks Artificial Intelligence against urologists in the context of Robot-Assisted Partial Nephrectomy (RAPN)—a complex, non-linear procedure. The study investigates the impact of temporal context on performance and evaluates the gap between human expertise and state-of-the-art neural networks.

Methods and Participants

The researchers conducted a comparative study involving a human cohort and an AI pipeline. The human survey recruited 100 participants across three levels of expertise: medical students, surgical trainees, and consultants.

Demographics and expertise levels of the 100 human participants.

Participants were tasked with identifying surgical phases from 17 random moments, presented first as single still frames and then as 10-second video snippets. Concurrently, three AI architectures were evaluated: a frame-based ResNet50, a temporal ResNet50-LSTM, and TeCNO, a multi-stage model designed for full workflow analysis.

Datasets

The study utilized two primary datasets:

  • Cholec80: A standard benchmark of 80 laparoscopic cholecystectomy videos used for model validation.
  • RAPN Dataset: A challenging dataset of 143 robotic partial nephrectomy recordings, featuring 15 distinct phases and a high degree of procedural non-linearity.

Results: The "Motion Premium"

A key finding was the significant performance boost provided by temporal context. For all human groups, moving from still frames to 10-second video snippets improved accuracy. Consultants, for instance, saw a 12% increase in performance when given motion cues.

Survey scores across professions (a) and statistical significance of the video-based performance boost (b).

Despite human expertise, the AI models demonstrated superior raw classification capabilities. The TeCNO model achieved an accuracy of 61.6%, notably higher than the 53.2% average achieved by consultants on video snippets. This suggests that while human expertise is grounded in anatomical knowledge, AI can leverage large-scale data to master complex visual patterns.

Key Insights

  • Shared Ambiguities: Both humans and AI struggled with specific phases like "Hilar Control," which lack distinct visual landmarks.
  • Data vs. Knowledge: AI performance dropped to student levels when trained on only 25% of the data, highlighting its reliance on volume compared to human anatomical reasoning.
  • Confidence Disconnect: While consultants expressed higher confidence than students, their subjective certainty did not always correlate with higher objective accuracy.

Conclusion and Future Work

The study concludes that while AI currently leads in raw accuracy for SPR in complex surgeries, both humans and machines benefit immensely from temporal context. Future research will explore longer context windows (minutes rather than seconds) and multi-task learning to better integrate tool and organ recognition into phase classification. Furthermore, the discrepancy between expert confidence and accuracy suggests a need for caution regarding confirmation bias in manual data annotation.

Summary updated

BibTeX
@article{mezzina2025surgeons,
  author = {Marco Mezzina and Pieter De Backer and Tom Vercauteren and Matthew B. Blaschko and Alexandre Mottrie and Tinne Tuytelaars},
  title = {{Surgeons vs. Computer Vision: A comparative analysis on surgical phase recognition capabilities}},
  journal = {Int. J. Comput. Assist. Radiol. Surg.},
  year = {2025},
  doi = {10.1007/s11548-025-03383-4},
}
Figure