Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgery

Chang Shi, Yi Zheng, Ann Majewicz Fey

TL;DR

This paper demonstrates that Transformer models, originally designed for natural language processing, can effectively recognize and predict surgical gestures and tool trajectories using only robot kinematic data. By ditching computationally expensive video processing, the authors achieve state-of-the-art accuracy (89.3%) and low trajectory error (2.71mm), moving us closer to real-time "intelligent" surgical assistants.

What Problem Does This Paper Address?

In Robot-Assisted Surgery (RAS), the goal is to move from simple teleoperation to "Robot-Enhanced Surgery" (RES), where the robot acts as a collaborative partner. To do this, the system must understand the context of the operation: what is the surgeon doing right now, and what are they likely to do in the next second?

Existing methods for surgical activity recognition often rely on a combination of video and kinematic data (the "telemetry" of the robot arms). However, processing video is computationally intensive and introduces latency, making real-time assistance difficult. Furthermore, traditional sequence models like LSTMs (Long Short-Term Memory) can struggle to capture long-range dependencies in complex surgical tasks. The authors tackle the challenge of performing recognition and prediction using only kinematic data while maintaining high accuracy.

Methods

The authors adapted the standard Transformer encoder-decoder architecture for three distinct tasks:

  1. Gesture Recognition: Identifying the current action (e.g., "positioning needle") based on current robot movements.
  2. Gesture Prediction: Anticipating the next gesture in the sequence.
  3. Trajectory Prediction: Forecasting the physical path ($x, y, z$ coordinates) of the robot's end-effectors.

Key Modifications to the Transformer: * Removed Embedding Layers: Since the input is continuous numerical kinematic data rather than discrete words, the standard NLP embedding layers were removed. * Removed Padding Masks: Because the authors used fixed-size observation windows, padding masks (used in NLP for varying sentence lengths) were unnecessary. * Encoder Output Layer: They added a fully connected layer to the encoder to map its output dimensions to the decoder's requirements, facilitating the attention mechanism between the two.

The model uses "Attention" to weigh the importance of different time steps in a sequence, allowing it to focus on critical movements that define a specific gesture.

Datasets

The study utilized the JIGSAWS (JHU-ISI Gesture and Skill Assessment Working Set) dataset.

  • Tasks: Specifically focused on 39 trials of the "Suturing" task.
  • Data: 38 kinematic features per frame, including end-effector positions, rotation matrices, linear/angular velocities, and gripper angles.
  • Ground Truth: 16 distinct gesture classes (e.g., G1: Reaching for needle, G3: Pushing needle through tissue).

Key Insights and Ideas

  • Kinematics are Sufficient: The most striking insight is that Transformers can outperform models that use both video and kinematics by using kinematics alone. This suggests that the "signature" of a surgical gesture is deeply embedded in the robot's motion data.
  • Attention over Recurrence: Unlike RNNs or LSTMs that process data step-by-step, the Transformer looks at the entire observation window at once. This global view helps it better understand the transition between complex surgical sub-tasks.
  • The "Translation" Analogy: The authors successfully treated gesture recognition as a "translation" problem—translating a sequence of physical movements into a sequence of semantic labels.

Key Results

The models were evaluated using Leave-One-User-Out (LOUO) cross-validation to ensure the results generalize to different surgeons:

  • Gesture Recognition: Achieved 89.3% accuracy, outperforming the "Fusion-KVE" method (86.3%) which requires video data.
  • Gesture Prediction: Achieved 84.6% accuracy for predicting the gesture 1 second into the future.
  • Trajectory Prediction: Achieved a Mean Absolute Error (MAE) of just 2.71mm for a 1-second horizon.

These results are competitive with or superior to state-of-the-art methods while being significantly more efficient due to the lack of video processing.

Limitations and Future Work

  • Independent Evaluation: Currently, the prediction models were tested using "ground truth" inputs for the previous steps. In a real-world deployment, the prediction model would rely on the output of the recognition model. If the recognition model makes a mistake, that error might propagate.
  • Ablation Studies: The authors suggest more work is needed to understand which specific gestures are harder to predict than others.
  • End-to-End Integration: Future work will focus on jointly evaluating these models to test their robustness in a truly autonomous pipeline, where estimated values are fed forward through the recognition and prediction stages.

Summary updated

Abstract

Surgical activity recognition and prediction can help provide important context in many Robot-Assisted Surgery (RAS) applications, for example, surgical progress monitoring and estimation, surgical skill evaluation, and shared control strategies during teleoperation. Transformer models were first developed for Natural Language Processing (NLP) to model word sequences and soon the method gained popularity for general sequence modeling tasks. In this paper, we propose the novel use of a Transformer model for three tasks: gesture recognition, gesture prediction, and trajectory prediction during RAS. We modify the original Transformer architecture to be able to generate the current gesture sequence, future gesture sequence, and future trajectory sequence estimations using only the current kinematic data of the surgical robot end-effectors. We evaluate our proposed models on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) and use Leave-One-User-Out (LOUO) cross validation to ensure generalizability of our results. Our models achieve up to 89.3% gesture recognition accuracy, 84.6% gesture prediction accuracy (1 second ahead) and 2.71mm trajectory prediction error (1 second ahead). Our models are comparable to and able to outperform state-of-the-art methods while using only the kinematic data channel. This approach can enable near-real time surgical activity recognition and prediction.
BibTeX
@inproceedings{shi2022recognition,
  author = {Chang Shi and Yi Zheng and Ann Majewicz Fey},
  title = {{Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgery}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=bbasMs4WwW},
}

Loading related content...

Figure