Recurrent and Spiking Modeling of Sparse Surgical Kinematics

Neil Getty, Zixuan Zhao, Stephan Gruessner, Liaohai Chen, Fangfang Xia

TL;DR

Researchers have demonstrated that "motion signatures" in robotic surgery are unique enough to identify specific surgeons and tasks using only sparse kinematic data—no video required. By converting these models into Spiking Neural Networks (SNNs), they achieved high accuracy with a framework ready for low-power, real-time deployment on neuromorphic hardware.

What Problem Does This Paper Address?

In the world of robot-assisted surgery, assessing a surgeon's skill usually involves analyzing both video feeds and kinematic logs (the data of how the robotic arms move). Most existing studies focus on distinguishing between "novices" and "experts," which is a relatively easy classification task.

However, this paper tackles a much harder problem: Can we distinguish between different surgeons who are all highly proficient? Furthermore, can we do it using only kinematic data, ignoring the data-heavy video feeds? Doing so would improve privacy, reduce latency, and allow for "edge" processing—where the AI runs directly on the surgical robot's hardware rather than in a distant cloud server.

Methods

The authors took a two-step approach to modeling surgical motion:

  1. Sparse Event Encoding: Instead of using raw positional data, they calculated the movement between timesteps. They then applied a threshold to turn these movements into binary "events" (spikes). If the movement was above a certain level, it triggered a '1'; otherwise, it was a '0'. This dramatically increases data sparsity, making it ideal for neuromorphic (brain-inspired) computing.
  2. Deep Learning & SNN Conversion:
    • They trained standard Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNNs) on this binary data.
  3. Using the Nengo simulation framework, they converted these traditional artificial neural networks into Spiking Neural Networks (SNNs). SNNs mimic biological neurons by communicating via discrete spikes, which is significantly more energy-efficient when run on specialized hardware like the Intel Loihi chip.

Datasets

The study introduces a new dataset created at the University of Illinois at Chicago (UIC):

  • Source: Data captured from a MIMIC FlexVR simulator.
  • Participants: 4 surgical fellows (all with similar, high levels of proficiency).
  • Tasks: 4 representative exercises: Pick and Place, Peg Board, Ring and Rail, and Thread the Needle.
  • Scale: 120 exercises, each containing roughly 4,000 timesteps of kinematic data (instrument positions, orientations, and gripper angles).

Key Insights and Ideas

  • The "Motion Signature": Even among experts, surgeons have unique "styles." The model was able to identify individual surgeons with high accuracy, suggesting that motion characteristics are as unique as a fingerprint.
  • Lossless SNN Conversion: One of the biggest hurdles in neuromorphic computing is the "accuracy tax" paid when converting a standard network to a spiking one. This paper found no noticeable loss in accuracy during conversion, proving that surgical kinematics are a natural fit for event-based modeling.
  • Left-Hand Dominance: Feature importance analysis revealed that "Pitch" and "Roll" of the left arm were the most critical features for classification. This is likely because tasks like Thread the Needle require complex, subtle rotations of the non-dominant hand to position the needle correctly.

Key Results

  • Surgeon Identification: The LSTM model achieved 83.43% accuracy in identifying which of the four fellows was performing the task.
  • Task Prediction: The model achieved 79.14% accuracy in identifying which of the four surgical exercises was being performed.
  • SNN Performance: The spiking versions of the models performed nearly identically to their non-spiking counterparts, validating the sparse encoding strategy.
  • LGBM Baseline: Traditional gradient boosting (LGBM) performed well on raw data but struggled with the binary event-encoded data, where the LSTM excelled.

Limitations and Future Work

  • Sequence Length: The current models use relatively short sequences. The authors suggest exploring Legendre Memory Units (LMUs) in the future to handle longer-term temporal associations.
  • Simulation vs. Reality: The data comes from a simulator. The next step is "domain adaptation"—taking models trained in simulation and applying them to data from real-world clinical robotic systems.
  • Hardware Deployment: While the models were tested in the Nengo simulator, the authors aim to move to physical neuromorphic chips (like Intel's Loihi) to measure actual power savings and latency improvements in a real-time surgical environment.

Summary updated

Abstract

Robot-assisted minimally invasive surgery is improving surgeon performance and patient outcomes. This innovation is also turning what has been a subjective practice into motion sequences that can be precisely measured. A growing number of studies have used machine learning to analyze video and kinematic data captured from surgical robots. In these studies, models are typically trained on benchmark datasets for representative surgical tasks to assess surgeon skill levels. While they have shown that novices and experts can be accurately classified, it is not clear whether machine learning can separate highly proficient surgeons from one another, especially without video data. In this study, we explore the possibility of using only kinematic data to predict surgeons of similar skill levels. We focus on a new dataset created from surgical exercises on a simulation device for skill training. A simple, efficient encoding scheme was devised to encode kinematic sequences so that they were amenable to edge learning. We report that it is possible to identify surgical fellows receiving near perfect scores in the simulation exercises based on their motion characteristics alone. Further, our model could be converted to a spiking neural network to train and infer on the Nengo simulation framework with no loss in accuracy. Overall, this study suggests that building neuromorphic models from sparse motion features may be a potentially useful strategy for identifying surgeons and gestures with chips deployed on robotic systems to offer adaptive assistance during surgery and training with additional latency and privacy benefits.
BibTeX
@inproceedings{getty2020recurrent,
  author = {Neil Getty and Zixuan Zhao and Stephan Gruessner and Liaohai Chen and Fangfang Xia},
  title = {{Recurrent and Spiking Modeling of Sparse Surgical Kinematics}},
  booktitle = {ICONS},
  year = {2020},
  url = {https://openreview.net/forum?id=B7GwKGaI2Ip},
}

Loading related content...

Figure