Surgeons versus computer vision: a comparative analysis on surgical phase recognition capabilities

Marco Mezzina, Pieter De Backer, Tom Vercauteren, Matthew B. Blaschko, Alexandre Mottrie, Tinne Tuytelaars

Introduction

The transition toward the "digital operating room" requires AI systems to possess a nuanced, real-time understanding of surgical workflows. This capability, known as Surgical Phase Recognition (SPR), is essential for context-aware assistance and automated documentation. While current computer vision models demonstrate high proficiency in linear procedures, such as laparoscopic cholecystectomies, they often struggle with the inherent complexity of non-linear surgeries.

Robot-Assisted Partial Nephrectomy (RAPN) serves as a prime example of such complexity, featuring 15 distinct phases that may occur out of sequence or vary significantly between patients. This study benchmarks human experts—ranging from medical students to consultant urologists—against state-of-the-art deep learning models to quantify the "performance gap" and identify the visual cues critical for accurate workflow tracking.

Methodology

The researchers implemented a comprehensive evaluation framework consisting of a human survey and an AI benchmarking pipeline.

Human Performance Survey

A custom web-based platform was developed to evaluate 100 participants. These individuals were categorized by their level of expertise: medical students, surgical trainees, and consultants. Participants were presented with either a static single frame or a 10-second video snippet from a RAPN procedure and tasked with identifying the active surgical phase.

Distribution of expertise levels among the 100 survey participants, showing a diverse range of clinical experience.

AI Pipeline and Benchmarking

Three distinct AI architectures were trained and tested on the same task:

  • ResNet50: A baseline image classification model that processes frames independently without temporal context.
  • ResNet50-LSTM: A hybrid model utilizing Long Short-Term Memory (LSTM) units to incorporate temporal information from 10-second and 60-second windows.
  • TeCNO: A multi-stage temporal convolutional network specifically designed to handle the long-range dependencies in surgical video.

Data Sources

The study utilized two primary datasets:

  1. Cholec80: A standard benchmark of 80 gallbladder removal videos used for initial model validation.
  2. RAPN Dataset: A challenging dataset comprising 143 full-length robot-assisted partial nephrectomy recordings. This dataset served as the primary "stress test" due to the non-linear nature of the procedure.

Analysis of Results

The Critical Role of Temporal Context

A primary finding was that temporal context significantly enhances recognition accuracy for both humans and machines. Moving from a single frame to a 10-second video snippet led to a marked improvement in performance across all expertise levels. Human participants reported using "Movements & Gestures" as a key disambiguator for phases that appear visually identical in static images.

Survey scores across professions, demonstrating the significant performance boost provided by 10-second video snippets compared to single frames.

Expertise and Shared Blind Spots

While consultants and trainees naturally outperformed medical students, the study revealed that even experts find certain phases inherently ambiguous. Interestingly, both human experts and AI models struggled with the same specific phases, such as "Hilar Control" and "Specimen Retrieval." This suggests that errors in SPR are often rooted in the visual complexity of the surgical site rather than purely algorithmic failures.

Comparative Performance Metrics

  • Human Accuracy: Consultant urologists achieved an average accuracy of 41.2% on single frames, which improved to 53.2% when provided with 10-second video snippets.
  • AI Accuracy: The top-performing AI model, TeCNO, achieved an accuracy of 61.4% on the RAPN dataset, notably exceeding the performance of human experts under the same restricted-context conditions.
  • Visual Cues: Surgeons identified surgical tools (92%) and anatomic structures (81%) as their primary points of focus. This confirms that future high-performance SPR models should likely employ multi-task learning to simultaneously track instruments and segment anatomy.

Conclusion and Future Directions

The study demonstrates that while AI models can match or exceed human performance in identifying surgical phases from short clips, human experts still possess a unique "prior knowledge" advantage. Surgeons leverage years of anatomical training to localize phases even when visual data is degraded.

Future research should focus on extending the temporal context of AI models beyond 60 seconds to resolve the most difficult procedural ambiguities. Furthermore, the integration of multi-task learning—combining tool tracking, organ segmentation, and phase recognition—appears to be the most promising path toward achieving human-level reasoning in the digital operating room.

Summary updated

BibTeX
@article{mezzina2025surgeons,
  author = {Marco Mezzina and Pieter De Backer and Tom Vercauteren and Matthew B. Blaschko and Alexandre Mottrie and Tinne Tuytelaars},
  title = {{Surgeons versus computer vision: a comparative analysis on surgical phase recognition capabilities}},
  journal = {Int. J. Comput. Assist. Radiol. Surg.},
  year = {2025},
  doi = {10.1007/s11548-025-03383-4},
}
Figure