Automatic assessment of robotic suturing utilizing computer vision in a dry-lab simulation

Sarah Choksi, Sanjeev Narasimhan, Mattia Ballo, Mehmet Kerem Türkcan, Yiran Hu, Chengbo Zang, Alex Farrell, B. King, Julian J. Nussbaum, Adin Reisner, Zoran Kostić, Giovanni Taffurelli, Filippo FilicoriArtificial 2025 · DOI

TL;DR

Researchers have developed a deep learning pipeline that automatically evaluates robotic suturing skills by analyzing video from dry-lab simulations. By combining Video Swin Transformers for action recognition with a Random Forest classifier, the system can distinguish between proficient surgeons and trainees with 66.7% accuracy.

What Problem Does This Paper Address?

As robot-assisted surgery becomes the standard of care, the bottleneck for training new surgeons remains the "expert-in-the-loop" requirement. Currently, surgical skill assessment is largely subjective, time-consuming, and dependent on the availability of senior surgeons to manually review videos.

While standardized curricula exist for laparoscopy (like the Fundamentals of Laparoscopic Surgery), robotic surgery lacks a universally validated, automated assessment framework. This paper aims to bridge that gap by creating a computer vision (CV) model that can provide objective, automated feedback on suturing—one of the most fundamental and difficult surgical skills to master.

Methods

The researchers developed a multi-stage deep learning architecture to analyze two specific tasks: backhand suturing and railroad suturing (a continuous running stitch).

  1. Data Pre-processing: Videos were recorded at 30 FPS and downsampled to 15 FPS. To capture long-term temporal context without overwhelming the GPU memory, the team used a dilation (frame skip) technique. Instead of looking at 16 consecutive frames (approx. 1 second), they sampled 16 frames with a stride of 15, creating an effective temporal window of ~17 seconds.
  2. Feature Extraction: They fine-tuned two separate Video Swin Transformer models.
  3. Model A classified the "sub-stitch phase" (Needle Positioning, Targeting, Driving, or Withdrawal).
  4. Model B predicted a binary "technical score" (Ideal vs. Non-ideal) for each action.
  5. Proficiency Classification: The outputs from these transformers were aggregated and fed into a Random Forest Classifier. This final stage determined if the operator was "Proficient" (defined as having >50 robotic cases of experience) or a "Trainee."

Datasets

The study utilized a custom dataset generated in a dry-lab setting:

  • Participants: 27 surgeons (20 trainees, 7 attendings).
  • Videos: 102 total videos (51 backhand, 51 railroad).
  • Annotations: A total of 862,038 frames were annotated by surgical residents. Each video was segmented into the four sub-stitch phases and labeled with technical quality scores.
  • Ground Truth: Proficiency was determined by case volume and confirmed via video review by two fellowship-trained attending surgeons.

Key Insights and Ideas

  • The "Needle Driving" Signature: Using Mean Decrease in Impurity (MDI) analysis, the researchers found that the "Needle Driving" phase was the most significant predictor of overall proficiency. How a surgeon handles the needle as it passes through tissue is the clearest "tell" of their experience level.
  • Temporal Context Matters: Standard video models often look at very short clips (1–2 seconds). By using dilated sampling to look at 17-second windows, the model could better understand the intent and flow of the suturing motion, which is critical for assessing skill.
  • Standardizing the "Surgical Grammar": The paper reinforces a "taxonomy" of suturing (Positioning -> Targeting -> Driving -> Withdrawal). Breaking complex tasks into these discrete "gestures" makes it easier for AI to provide actionable feedback (e.g., "You are struggling specifically with needle targeting").

Key Results

  • Sub-stitch Phase Classification: Achieved an average accuracy of 70.23%.
  • Technical Score Prediction: Achieved an average accuracy of 68.4%.
  • Surgeon Proficiency Prediction: The combined Random Forest model achieved 66.7% accuracy in identifying whether the surgeon was an expert or a trainee.
  • False Negatives: Only six participants were incorrectly labeled as trainees when they were actually proficient, suggesting the model is relatively conservative in its "expert" designations.

Limitations and Future Work

  • Sample Size: With only 27 participants, the model's ability to generalize is limited. The authors note that a larger, more diverse dataset is needed to improve accuracy.
  • Dry-Lab Constraints: The simulation used artificial skin and a bench-top model. Real-world surgery involves blood, varying tissue textures, and camera movement, which would significantly increase the complexity for a CV model.
  • Object Recognition: Future iterations could benefit from explicit object tracking (e.g., tracking the needle tip and tool coordinates) rather than relying solely on whole-frame video transformers.
  • Feedback Loop: The current model classifies proficiency but doesn't yet provide real-time, "how-to-improve" coaching to the trainee.

Summary updated

Abstract

Aim: Automated surgical skill assessment is poised to become an invaluable asset in surgical residency training. In our study, we aimed to create deep learning (DL) computer vision artificial intelligence (AI) models capable of automatically assessing trainee performance and determining proficiency on robotic suturing tasks. Methods: Participants performed two robotic suturing tasks on a bench-top model created by our lab. Videos were recorded of each surgeon performing a backhand suturing task and a railroad suturing task at 30 frames per second (FPS) and downsampled to 15 FPS for the study. Each video was segmented into four sub-stitch phases: needle positioning, targeting, driving, and withdrawal. Each sub-stitch was annotated with a binary technical score (ideal or non-ideal), reflecting the operator’s skill while performing the suturing action. For DL analysis, 16-frame overlapping clips were sampled from the videos with a stride of 1. To extract the features useful for classification, two pretrained Video Swin Transformer models were fine-tuned using these clips: one to classify the sub-stitch phase and another to predict the technical score. The model outputs were then combined and used to train a Random Forest Classifier to predict the surgeon's proficiency level. Results: A total of 102 videos from 27 surgeons were evaluated using 3-fold cross-validation, 51 videos for the backhand suturing task and 51 videos for the railroad suturing task. Performance was assessed on sub-stitch classification accuracy, technical score accuracy, and surgeon proficiency prediction. The clip-based Video Swin Transformer models achieved an average classification accuracy of 70.23% for sub-stitch classification and 68.4% for technical score prediction on the test folds. Combining the model outputs, the Random Forest Classifier achieved an average accuracy of 66.7% in predicting surgeon proficiency. Conclusion: This study shows the feasibility of creating a DL-based automatic assessment tool for robotic-assisted surgery. Using machine learning models, we predicted the proficiency level of a surgeon with 66.7% accuracy. Our dry lab model proposes a standardized training and assessment tool for suturing tasks using computer vision.
BibTeX
@inproceedings{choksi2025automatic,
  author = {Sarah Choksi and Sanjeev Narasimhan and Mattia Ballo and Mehmet Kerem Türkcan and Yiran Hu and Chengbo Zang and Alex Farrell and B. King and Julian J. Nussbaum and Adin Reisner and Zoran Kostić and Giovanni Taffurelli and Filippo Filicori},
  title = {{Automatic assessment of robotic suturing utilizing computer vision in a dry-lab simulation}},
  booktitle = {Artificial},
  year = {2025},
  doi = {10.20517/ais.2024.84},
}
Figure