Video-based surgical skill assessment using 3D convolutional neural networks

Isabel Funke, Sören Torge Mees, Jürgen Weitz, Stefanie Speidel

Abstract

A profound education of novice surgeons is crucial to ensure that surgical interventions are effective and safe. One important aspect is the teaching of technical skills for minimally invasive or robot-assisted procedures. This includes the objective and preferably automatic assessment of surgical skill. Recent studies presented good results for automatic, objective skill evaluation by collecting and analyzing motion data such as trajectories of surgical instruments. However, obtaining the motion data generally requires additional equipment for instrument tracking or the availability of a robotic surgery system to capture kinematic data. In contrast, we investigate a method for automatic, objective skill assessment that requires video data only. This has the advantage that video can be collected effortlessly during minimally invasive and robot-assisted training scenarios. Our method builds on recent advances in deep learning-based video classification. Specifically, we propose to use an inflated 3D ConvNet to classify snippets, i.e., stacks of a few consecutive frames, extracted from surgical video. The network is extended into a Temporal Segment Network during training. Our approach achieves high skill classification accuracies ranging from 95.1% to 100.0%.

Introduction

Objective assessment of surgical skill is vital for training proficient surgeons. Traditionally, this relies on expert observation using standardized checklists, a process that is both time-consuming and prone to subjectivity. While automated systems (Objective Computer-Aided Technical Skill Evaluation, or OCASE-T) have emerged, they typically rely on kinematic data—such as instrument trajectories and velocities—captured via specialized tracking hardware or robotic sensors.

The goal of this research is to develop a system capable of assessing skill using only the standard video feed from laparoscopic or robotic cameras. Video data is easily accessible but presents significant analytical challenges due to its high dimensionality. This paper explores the use of 3D convolutional neural networks to extract spatiotemporal patterns directly from raw video to distinguish between novice, intermediate, and expert surgeons.

Methods

The proposed approach leverages three key advancements in computer vision to analyze surgical video:

  1. Inflated 3D ConvNets (I3D): Unlike standard 2D networks, I3D uses an "inflated" Inception-v1 architecture. This allows filters to learn across both spatial and temporal dimensions, capturing the motion "flow" of surgical gestures.
  2. Temporal Segment Networks (TSN): To process long video sequences efficiently, the TSN framework divides videos into segments and samples short snippets (64 frames each). During training, a consensus mechanism aggregates information from multiple snippets to determine the overall skill level.
  3. Cross-Domain Pretraining: Given the limited size of surgical datasets, the model was pretrained on Kinetics, a large-scale dataset of human actions. This allows the network to learn general motion features before being fine-tuned on surgical tasks.
Architecture of the proposed 3D ConvNet and Temporal Segment Network framework.

The system was evaluated on the JIGSAWS dataset, which includes recordings of three basic robot-assisted tasks: Suturing, Needle Passing, and Knot Tying. Eight participants with varying experience levels (Novice, Intermediate, Expert) performed these tasks on a dry-lab bench-top model.

Results and Discussion

The model demonstrated exceptional performance using a "Leave-one-supertrial-out" (LOSO) cross-validation scheme. For the Suturing task, the system achieved 100% accuracy using both RGB video and Optical Flow. In Needle Passing, accuracy reached 100% with Optical Flow and 96.4% with RGB. Knot Tying proved the most challenging, with an accuracy of approximately 95.8%.

Comparison of classification performance across different methods and tasks.

Key findings include:

  • End-to-End Learning: The 3D ConvNet successfully learned discriminative patterns directly from pixels, removing the need for manual feature engineering (e.g., path length calculations).
  • Transfer Learning: Pretraining on the Kinetics dataset was crucial; without it, accuracy dropped significantly to 41%, proving that general motion knowledge is highly transferable to the surgical domain.
  • Consensus Mechanism: The TSN framework effectively resolved contradictions by aggregating data across the entire performance rather than relying on isolated snippets.

Conclusion

This study demonstrates the feasibility of using deep learning to assess surgical technical skill directly from video. The 3D ConvNet approach achieves results comparable to or better than methods utilizing kinematic data. However, challenges remain regarding data diversity; performance decreased during "Leave-one-user-out" validation, suggesting the need for larger, more diverse datasets. Future work will focus on modeling long-term temporal relations and validating the system on complex intraoperative footage involving real-world variables like smoke and varying lighting.

Summary updated

Abstract

Purpose: A profound education of novice surgeons is crucial to ensure that surgical interventions are effective and safe. One important aspect is the teaching of technical skills for minimally invasive or robot-assisted procedures. This includes the objective and preferably automatic assessment of surgical skill. Recent studies presented good results for automatic, objective skill evaluation by collecting and analyzing motion data such as trajectories of surgical instruments. However, obtaining the motion data generally requires additional equipment for instrument tracking or the availability of a robotic surgery system to capture kinematic data. In contrast, we investigate a method for automatic, objective skill assessment that requires video data only. This has the advantage that video can be collected effortlessly during minimally invasive and robot-assisted training scenarios. Methods: Our method builds on recent advances in deep learning-based video classification. Specifically, we propose to use an inflated 3D ConvNet to classify snippets, i.e., stacks of a few consecutive frames, extracted from surgical video. The network is extended into a Temporal Segment Network during training. Results: We evaluate the method on the publicly available JIGSAWS dataset, which consists of recordings of basic robot-assisted surgery tasks performed on a dry lab bench-top model. Our approach achieves high skill classification accuracies ranging from 95.1% to 100.0%. Conclusions: Our results demonstrate the feasibility of deep learning-based assessment of technical skill from surgical video. Notably, the 3D ConvNet is able to learn meaningful patterns directly from the data, alleviating the need for manual feature engineering. Further evaluation will require more annotated data for training and testing.
BibTeX
@article{funke2019video,
  author = {Isabel Funke and Sören Torge Mees and Jürgen Weitz and Stefanie Speidel},
  title = {{Video-based surgical skill assessment using 3D convolutional neural networks}},
  journal = {Int. J. Comput. Assist. Radiol. Surg.},
  year = {2019},
  url = {https://openreview.net/forum?id=wz1CHHpcGSVA},
}
Figure