TL;DR
This paper introduces a novel method for automatically assessing surgical skill during unstructured procedures like nasal septoplasty, using only the motion of surgical tools. By analyzing tool movements into two distinct activity types, they achieve a respectable 72% accuracy in distinguishing between expert and novice surgeons, offering a path towards personalized, actionable feedback for trainees.
What Problem Does This Paper Address?
Traditional surgical training heavily relies on subjective feedback from attending surgeons. While objective skill assessment methods exist, they often focus on highly structured procedures (like cholecystectomy) and use generic metrics (time, movement count) that offer limited insight for improvement. Unstructured procedures, such as nasal septoplasty, present a significant challenge because they lack a clear, sequential structure, and attending surgeons often cannot directly observe the fine details of tool manipulation within the confined surgical field. This makes it difficult to provide trainees with specific, actionable feedback on their technical proficiency, hindering efficient learning.
Methods
The authors tackle this by developing an automated system to analyze the unstructured tool motion during nasal septoplasty. Their approach involves:
- Automated Stroke Segmentation: They define a "stroke" as a continuous motion of the Cottle elevator, identifying its start and end points based on the tool's distance from the estimated septal plane.
- Defining Two Activity Types:
- Type I Activity (Brushing): Characterized by movements away from the septum, this reflects the surgeon's wrist motion consistency and the force applied to break adhesions.
- Type II Activity (Coverage): This describes the surgeon's pattern of movement across the septal plane, indicating their efficiency in covering the surgical area.
- Feature Extraction: Based on these activity types, they derive three key features:
- Stroke Curvature Consistency (SCC): Measures how consistent the curvature of individual strokes is. Experts are hypothesized to have lower SCC due to more regular motions.
- Stroke Duration Consistency (SDC): Assesses the variability in the duration of strokes.
- Coverage Rate (CR): Quantifies how much of the septal area is covered per stroke, reflecting surgical efficiency.
- Skill Classification: A Support Vector Machine (SVM) classifier is trained using these features to distinguish between expert and novice surgeons.
Datasets
The study utilized data collected from 48 nasal septoplasty cases performed at four hospital sites over one year. This data included:
- Tool Motion Data: Captured using an electromagnetic sensor attached to the Cottle elevator.
- Video Data: Recorded using Kinects for potential annotation and context.
- Ground Truth Labels: Based on the experience level of the surgeons, categorizing them as either "expert" (attending surgeons and fellows) or "novice" (resident surgeons).
The paper also mentions using a Hidden Markov Model (HMM) approach for comparison, referencing prior work that used similar data collection methods.
Key Insights and Ideas
- Unstructured Procedures are Assessable: The core insight is that even complex, unstructured surgical tasks can be objectively assessed by analyzing the subtle nuances of tool motion, moving beyond generic metrics.
- Two Key Activity Types Capture Skill: The identification of Type I (brushing/force) and Type II (coverage/search) activities provides a meaningful framework for understanding surgical technique in septoplasty. This is a novel way to decompose the procedure's motion.
- Feature Engineering is Crucial: The derived features (SCC, SDC, CR) are not just arbitrary measurements; they are hypothesized to directly reflect aspects of surgical skill (consistency, efficiency, adaptability) that are clinically relevant.
- Actionable Feedback Potential: The features offer the potential for personalized feedback. For instance, a high SCC might suggest a need to practice smoother wrist movements, while a low CR could indicate inefficient coverage strategies.
Key Results
- Classification Accuracy: The SVM classifier achieved an overall micro-average accuracy of 69.6% for leave-one-trial-out (TO) and 69.6% for leave-one-user-out (UO) setups. The macro-average accuracy was 72.1% for TO and 73.3% for UO.
- Feature Performance: Individual features showed comparable performance to using all three combined, suggesting that each feature captures significant discriminatory information.
- Expert vs. Novice Differences: Exploratory analysis revealed that experts demonstrated decreased SCC (more consistent motion) and higher CR (faster coverage) compared to novices. Interestingly, experts showed higher SDC, which the authors hypothesize might be due to their ability to adapt to patient anatomy.
- Comparison to HMM: The proposed method achieved more uniform performance across different testing setups (TO vs. UO) compared to an HMM approach, which showed a significant drop in accuracy for the TO setup.
Limitations and Future Work
- Ground Truth Subjectivity: The study acknowledges that the "ground truth" skill level was based on self-proclaimed experience (attending/fellow vs. resident). A more robust approach would involve a subjective grading system by expert raters.
- Limited Scope: The study is focused solely on nasal septoplasty. Future work could explore the applicability of these features to other unstructured surgical procedures, such as laparoscopic dissections.
- Improving Accuracy: The authors suggest supplementing data with more surgeons and exploring additional features to improve classification accuracy.
- Real-time Feedback: A key future direction is to evaluate the impact of providing real-time feedback based on these features on trainee skill acquisition.
- Subjective Grading System: Implementing a case-by-case rating system as ground truth is also a proposed future step.
Summary updated