Automated Objective Surgical Skill Assessment in the Operating Room Using Unstructured Tool Motion

Piyush Poddar, Narges Ahmidi, S. Swaroop Vedula, Lisa Ishii, Gregory D. Hager, Masaru IshiiarXiv 2014 · OpenReview

TL;DR

This paper introduces a novel method for automatically assessing surgical skill during unstructured procedures like nasal septoplasty, using only the motion of surgical tools. By analyzing tool movements into two distinct activity types, they achieve a respectable 72% accuracy in distinguishing between expert and novice surgeons, offering a path towards personalized, actionable feedback for trainees.

What Problem Does This Paper Address?

Traditional surgical training heavily relies on subjective feedback from attending surgeons. While objective skill assessment methods exist, they often focus on highly structured procedures (like cholecystectomy) and use generic metrics (time, movement count) that offer limited insight for improvement. Unstructured procedures, such as nasal septoplasty, present a significant challenge because they lack a clear, sequential structure, and attending surgeons often cannot directly observe the fine details of tool manipulation within the confined surgical field. This makes it difficult to provide trainees with specific, actionable feedback on their technical proficiency, hindering efficient learning.

Methods

The authors tackle this by developing an automated system to analyze the unstructured tool motion during nasal septoplasty. Their approach involves:

  • Automated Stroke Segmentation: They define a "stroke" as a continuous motion of the Cottle elevator, identifying its start and end points based on the tool's distance from the estimated septal plane.
  • Defining Two Activity Types:
    • Type I Activity (Brushing): Characterized by movements away from the septum, this reflects the surgeon's wrist motion consistency and the force applied to break adhesions.
  • Type II Activity (Coverage): This describes the surgeon's pattern of movement across the septal plane, indicating their efficiency in covering the surgical area.
  • Feature Extraction: Based on these activity types, they derive three key features:
  • Stroke Curvature Consistency (SCC): Measures how consistent the curvature of individual strokes is. Experts are hypothesized to have lower SCC due to more regular motions.
  • Stroke Duration Consistency (SDC): Assesses the variability in the duration of strokes.
  • Coverage Rate (CR): Quantifies how much of the septal area is covered per stroke, reflecting surgical efficiency.
  • Skill Classification: A Support Vector Machine (SVM) classifier is trained using these features to distinguish between expert and novice surgeons.

Datasets

The study utilized data collected from 48 nasal septoplasty cases performed at four hospital sites over one year. This data included:

  • Tool Motion Data: Captured using an electromagnetic sensor attached to the Cottle elevator.
  • Video Data: Recorded using Kinects for potential annotation and context.
  • Ground Truth Labels: Based on the experience level of the surgeons, categorizing them as either "expert" (attending surgeons and fellows) or "novice" (resident surgeons).

The paper also mentions using a Hidden Markov Model (HMM) approach for comparison, referencing prior work that used similar data collection methods.

Key Insights and Ideas

  • Unstructured Procedures are Assessable: The core insight is that even complex, unstructured surgical tasks can be objectively assessed by analyzing the subtle nuances of tool motion, moving beyond generic metrics.
  • Two Key Activity Types Capture Skill: The identification of Type I (brushing/force) and Type II (coverage/search) activities provides a meaningful framework for understanding surgical technique in septoplasty. This is a novel way to decompose the procedure's motion.
  • Feature Engineering is Crucial: The derived features (SCC, SDC, CR) are not just arbitrary measurements; they are hypothesized to directly reflect aspects of surgical skill (consistency, efficiency, adaptability) that are clinically relevant.
  • Actionable Feedback Potential: The features offer the potential for personalized feedback. For instance, a high SCC might suggest a need to practice smoother wrist movements, while a low CR could indicate inefficient coverage strategies.

Key Results

  • Classification Accuracy: The SVM classifier achieved an overall micro-average accuracy of 69.6% for leave-one-trial-out (TO) and 69.6% for leave-one-user-out (UO) setups. The macro-average accuracy was 72.1% for TO and 73.3% for UO.
  • Feature Performance: Individual features showed comparable performance to using all three combined, suggesting that each feature captures significant discriminatory information.
  • Expert vs. Novice Differences: Exploratory analysis revealed that experts demonstrated decreased SCC (more consistent motion) and higher CR (faster coverage) compared to novices. Interestingly, experts showed higher SDC, which the authors hypothesize might be due to their ability to adapt to patient anatomy.
  • Comparison to HMM: The proposed method achieved more uniform performance across different testing setups (TO vs. UO) compared to an HMM approach, which showed a significant drop in accuracy for the TO setup.

Limitations and Future Work

  • Ground Truth Subjectivity: The study acknowledges that the "ground truth" skill level was based on self-proclaimed experience (attending/fellow vs. resident). A more robust approach would involve a subjective grading system by expert raters.
  • Limited Scope: The study is focused solely on nasal septoplasty. Future work could explore the applicability of these features to other unstructured surgical procedures, such as laparoscopic dissections.
  • Improving Accuracy: The authors suggest supplementing data with more surgeons and exploring additional features to improve classification accuracy.
  • Real-time Feedback: A key future direction is to evaluate the impact of providing real-time feedback based on these features on trainee skill acquisition.
  • Subjective Grading System: Implementing a case-by-case rating system as ground truth is also a proposed future step.

Summary updated

Abstract

Previous work on surgical skill assessment using intraoperative tool motion in the operating room (OR) has focused on highly-structured surgical tasks such as cholecystectomy. Further, these methods only considered generic motion metrics such as time and number of movements, which are of limited instructive value. In this paper, we developed and evaluated an automated approach to the surgical skill assessment of nasal septoplasty in the OR. The obstructed field of view and highly unstructured nature of septoplasty precludes trainees from efficiently learning the procedure. We propose a descriptive structure of septoplasty consisting of two types of activity: (1) brushing activity directed away from the septum plane characterizing the consistency of the surgeon's wrist motion and (2) activity along the septal plane characterizing the surgeon's coverage pattern. We derived features related to these two activity types that classify a surgeon's level of training with an average accuracy of about 72%. The features we developed provide surgeons with personalized, actionable feedback regarding their tool motion.
BibTeX
@misc{poddar2014automated,
  author = {Piyush Poddar and Narges Ahmidi and S. Swaroop Vedula and Lisa Ishii and Gregory D. Hager and Masaru Ishii},
  title = {{Automated Objective Surgical Skill Assessment in the Operating Room Using Unstructured Tool Motion}},
  year = {2014},
  eprint = {1412.6163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1412.6163},
}

Loading related content...

Figure