Video-based surgical skill assessment using 3D convolutional neural networks

Isabel Funke, Sören Torge Mees, Jürgen Weitz, Stefanie Speidel

TL;DR

This paper presents a deep learning approach to automatically assess surgical skills using only video data, eliminating the need for expensive motion-tracking hardware. By combining 3D Convolutional Neural Networks (I3D) with a Temporal Segment Network (TSN) framework, the researchers achieved skill classification accuracies between 95% and 100% on standard surgical tasks.

What Problem Does This Paper Address?

Training the next generation of surgeons requires objective, standardized feedback. Traditionally, this is done by experts using checklists, which is both time-consuming and subjective. While researchers have developed automated systems (Objective Computer-Aided Technical Skill Evaluation, or OCASE-T), most rely on kinematic data—trajectories, velocities, and angles captured by robotic sensors or specialized electromagnetic trackers.

The "holy grail" is a system that works using only the video feed from a standard laparoscopic or robotic camera. Video is effortless to collect but notoriously difficult to analyze because it is high-dimensional and complex. This paper tackles the challenge of extracting meaningful "spatiotemporal" patterns (how things move over time) directly from raw video to distinguish between novice, intermediate, and expert surgeons.

Methods

The authors bridge the gap between computer vision and surgery by adapting three major advances in action recognition:

  1. Inflated 3D ConvNets (I3D): Instead of using standard 2D networks that look at static frames, they use an "inflated" version of the Inception-v1 architecture. This allows the filters to learn across the temporal dimension, capturing the "flow" of a surgical gesture.
  2. Temporal Segment Networks (TSN): To handle long videos without overwhelming the GPU, the TSN framework divides a video into segments and samples short "snippets" (64 frames each). During training, the network acts like a Siamese architecture, looking at multiple snippets to reach a consensus on the surgeon's skill level.
  3. Cross-Domain Pretraining: Since surgical datasets are small, the authors used a model pretrained on Kinetics, a massive dataset of everyday human actions (like "playing guitar" or "shaking hands"). They then fine-tuned this model on surgical tasks.

The system was tested using two input modalities: standard RGB video and Optical Flow (which explicitly calculates the motion of pixels between frames).

Datasets

The study utilized the JIGSAWS (JHU-ISI Gesture and Skill Assessment Working Set) dataset. This is a benchmark in the field containing:

  • Tasks: Suturing, Needle Passing, and Knot Tying.
  • Setup: Performed on a dry-lab bench-top model using a da Vinci Surgical System.
  • Subjects: Eight participants with varying levels of experience, classified as Novice, Intermediate, or Expert.

Key Insights and Ideas

  • End-to-End Learning Wins: Unlike previous methods that required manual "feature engineering" (calculating path length or motion smoothness), the 3D ConvNet learns these discriminative patterns directly from the pixels.
  • General Motion Knowledge Transfers: One of the most surprising findings is that a network trained to recognize people "diving" or "making tea" (Kinetics) provides an excellent foundation for recognizing the nuances of "needle passing." Without this pretraining, the model's accuracy plummeted to 41%.
  • The "Knot Tying" Challenge: Knot tying proved to be the most difficult task to classify. The authors suggest this is because intermediate surgeons often exhibit "novice-like" motion patterns during this specific, highly technical task.
  • Consensus is Key: By using the TSN framework, the model avoids making a snap judgment based on a single 6-second snippet. Instead, it aggregates information from across the entire performance to resolve contradictions.

Key Results

The model was evaluated using a "Leave-one-supertrial-out" (LOSO) cross-validation scheme:

  • Suturing: 100% accuracy for both RGB and Optical Flow.
  • Needle Passing: 100% accuracy using Optical Flow; 96.4% using RGB.
  • Knot Tying: ~95.8% accuracy.

These results are on par with, or exceed, state-of-the-art methods that have the "unfair advantage" of using robotic kinematic data.

Limitations and Future Work

  • The "Data Hunger" Problem: While the results on the JIGSAWS dataset are impressive, the dataset only includes eight subjects. When the authors tried "Leave-one-user-out" (LOUO) validation, the performance dropped significantly because the model hadn't seen enough diverse surgical styles to generalize to a brand-new person.
  • Short-Term Memory: The current 3D ConvNet only "sees" snippets of about 6 seconds. It cannot yet model long-term temporal relations or the overall "economy of motion" over a 10-minute procedure.
  • Real-World Complexity: The JIGSAWS dataset is recorded in a controlled "dry lab." Future work needs to validate these models on real intraoperative footage, where blood, smoke, and varying lighting conditions make video analysis much harder.

Summary updated

Abstract

Purpose A profound education of novice surgeons is crucial to ensure that surgical interventions are effective and safe. One important aspect is the teaching of technical skills for minimally invasive or robot-assisted procedures. This includes the objective and preferably automatic assessment of surgical skill. Recent studies presented good results for automatic, objective skill evaluation by collecting and analyzing motion data such as trajectories of surgical instruments. However, obtaining the motion data generally requires additional equipment for instrument tracking or the availability of a robotic surgery system to capture kinematic data. In contrast, we investigate a method for automatic, objective skill assessment that requires video data only. This has the advantage that video can be collected effortlessly during minimally invasive and robot-assisted training scenarios. Methods Our method builds on recent advances in deep learning-based video classification. Specifically, we propose to use an inflated 3D ConvNet to classify snippets, i.e., stacks of a few consecutive frames, extracted from surgical video. The network is extended into a temporal segment network during training. Results We evaluate the method on the publicly available JIGSAWS dataset, which consists of recordings of basic robot-assisted surgery tasks performed on a dry lab bench-top model. Our approach achieves high skill classification accuracies ranging from 95.1 to 100.0%. Conclusions Our results demonstrate the feasibility of deep learning-based assessment of technical skill from surgical video. Notably, the 3D ConvNet is able to learn meaningful patterns directly from the data, alleviating the need for manual feature engineering. Further evaluation will require more annotated data for training and testing.
BibTeX
@article{funke2019video,
  author = {Isabel Funke and Sören Torge Mees and Jürgen Weitz and Stefanie Speidel},
  title = {{Video-based surgical skill assessment using 3D convolutional neural networks}},
  journal = {Int. J. Comput. Assist. Radiol. Surg.},
  year = {2019},
  url = {https://openreview.net/forum?id=wz1CHHpcGSVA},
}

Loading related content...

Figure