Vision Based Automated Surgical Skill Assessment

26 concepts23 papers1 proposal

Surgical training has traditionally relied on an apprenticeship model where senior surgeons observe trainees and provide qualitative feedback. While standardized rating scales attempt to formalize this process, human-centric approaches remain inherently subjective, time-consuming, and difficult to scale across large medical institutions (Towards Unified Surgical Skill Assessment). As argued by Sanchit Hira et al. in Video-based assessment of intraoperative surgical skill, manual video review introduces potential bias and limits the frequency of feedback provided to trainees. To overcome this bottleneck, a growing body of research focuses on automating surgical skill assessment by translating raw intraoperative video and sensor data into objective, reproducible performance metrics.

Deconstructing Surgical Motion into Kinematic Metrics

How do we convert the fluid, qualitative expertise of a surgeon into concrete data? The most direct approach focuses on tracking surgical instruments to measure "economy of motion."

By utilizing region-based convolutional neural networks (Faster R-CNNs) for spatial localization, Amy Jin et al. demonstrated in Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks that tracking tool usage patterns can objectively quantify technical proficiency. Expert surgeons move with high efficiency, minimizing redundant paths, whereas novices exhibit erratic and prolonged tool trajectories. Similarly, Hei Law et al. leveraged Stacked Hourglass Networks in Surgeon Technical Skill Assessment using Computer Vision based Analysis to track robotic instrument tips, feeding these kinematic trajectories into a linear classifier to accurately distinguish between high- and low-skill practitioners.

Beyond coordinate tracking, the underlying structure of surgical movement can be analyzed like a language. In String Motif-Based Description of Tool Motion for Detecting Skill and Gestures in Robotic Surgery, Narges Ahmidi et al. converted continuous 3D tool trajectories into discrete "coded strings." By identifying recurring patterns—or string motifs—the system simultaneously recognizes specific surgical gestures and evaluates the surgeon's skill level based on the syntactic efficiency of their movements.

Capturing Spatiotemporal Context with Deep Learning

While tracking tool tips provides a baseline for motion economy, surgery is a dynamic process where the interaction between tools and anatomy matters. Simple frame-by-frame analysis fails to capture the continuity and context of a procedure.

To capture these complex dynamics, modern frameworks leverage advanced temporal modeling. For instance, the model developed by Sanchit Hira et al. employs "dual attention" mechanisms that focus on both spatial features (where the tools interact with tissue) and temporal features (how those interactions evolve over time). This allows the network to outperform traditional hand-crafted features on real-world intraoperative videos.

More recently, the field has shifted toward end-to-end architectures like Video Swin Transformers. In AIxSuture: vision-based assessment of open suturing skills, Hanna Hoffmann et al. applied this approach to open surgery, achieving skill evaluations comparable to human expert panels. Similarly, Sarah Choksi et al. demonstrated in Automatic assessment of robotic suturing utilizing computer vision in a dry-lab simulation that combining Video Swin Transformers with a Random Forest classifier successfully distinguishes proficient robotic surgeons from trainees by processing raw video volumes directly, bypassing the need for explicit tool tracking hardware.

Contextual and Workflow Awareness

Technical skill is not merely about moving tools efficiently; it also requires maintaining a safe operative environment and executing a logical sequence of tasks.

One key indicator of surgical proficiency is how well a surgeon manages the surgical field. In Surgical Skill Assessment on In-Vivo Clinical Data via the Clearness of Operating Field, Daochang Liu et al. introduced the "clearness of the operating field" as an explicit metric for in-vivo clinical data. Expert surgeons actively manage bleeding and debris to maintain visibility, whereas novices often struggle with field clarity—a nuance that benchtop simulations fail to capture.

Furthermore, automated systems must understand the broader procedural context. In Novel evaluation of surgical activity recognition models using task-based efficiency metrics, Aneeq Zia et al. introduced a CNN-LSTM model to segment robotic prostatectomies into distinct phases, demonstrating that task-based efficiency metrics correlate strongly with expert human ratings. Taking this a step further, Jianfeng Cao et al. developed the AI-Endo framework in Intelligent surgical workflow recognition for endoscopic submucosal dissection with real-time animal study. By training on long-term data from a single expert surgeon to establish a gold-standard workflow benchmark, the authors enabled the automated generation of skill assessment reports that accurately capture trainee hesitation and decision-making fluidity.

Expanding the Sensing Modality

While video remains the primary data source, researchers are incorporating alternative sensing modalities to capture cognitive and structural dimensions of surgical performance.

To detect when a trainee is experiencing high cognitive load or operational difficulty, Bin Zheng et al. integrated eye-tracking data with LSTM networks in Detecting performance difficulty of learners in colonoscopy: Evidence from eye-tracking, identifying exactly when a learner "gets lost" during simulated colonoscopies. In the domain of reconstructive surgery, Matthew Shreve et al. utilized optical flow in Evaluation of Facial Reconstructive Surgery on Patients with Facial Palsy Using Optical Strain to calculate "optical strain," providing a non-invasive, marker-less method to objectively quantify post-surgical facial deformation. Finally, Philipp Stefan et al. demonstrated in Computer-assisted simulated workplace-based assessment in surgery: application of the universal framework of intraoperative performance within a mixed-reality simulation that embedding these automated assessment frameworks within mixed-reality simulations allows for a holistic evaluation of a practitioner's readiness in complex, team-based environments.

Key Foundational Work

Validating Metrics for a Mastoidectomy Simulator (2007) — contributed 2 concept elements to this direction.
Evaluation of Facial Reconstructive Surgery on Patients with Facial Palsy Using Optical Strain (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_subjective_surgical_skill_assessment_bc242ede,
  title = {subjective surgical skill assessment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subjective-surgical-skill-assessment/rdr_bc242edebeeb46ff90ba74211f21586c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_subjective_surgical_skill_assessment_bc242ede_v1,
  title = {subjective surgical skill assessment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subjective-surgical-skill-assessment/rdr_bc242edebeeb46ff90ba74211f21586c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake