Abstract
Surgical activity recognition and prediction can help provide important context in many Robot-Assisted Surgery (RAS) applications, for example, surgical progress monitoring and estimation, surgical skill evaluation, and shared control strategies during teleoperation. Transformer models were first developed for Natural Language Processing (NLP) to model word sequences and soon the method gained popularity for general sequence modeling tasks. In this paper, we propose the novel use of a Transformer model for three tasks: gesture recognition, gesture prediction, and trajectory prediction during RAS. We modify the original Transformer architecture to be able to generate the current gesture sequence, future gesture sequence, and future trajectory sequence estimations using only the current kinematic data of the surgical robot end-effectors. We evaluate our proposed models on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) and use Leave-One-User-Out (LOUO) cross-validation to ensure the generalizability of our results. Our models achieve up to 89.3% gesture recognition accuracy, 84.6% gesture prediction accuracy (1 second ahead) and 2.71mm trajectory prediction error (1 second ahead). Our models are comparable to and able to outperform state-of-the-art methods while using only the kinematic data channel. This approach can enable near-real time surgical activity recognition and prediction.
Introduction
The evolution of Robot-Assisted Surgery (RAS) aims to transition from simple teleoperation—where the robot acts as a passive extension of the surgeon—to "Robot-Enhanced Surgery" (RES). In this collaborative paradigm, the robotic system acts as an intelligent partner. To achieve this, the system must possess contextual awareness: it must recognize what the surgeon is doing in the moment and anticipate their next actions.
Traditionally, surgical activity recognition has relied heavily on video data or a combination of video and robot kinematics. However, video processing is computationally expensive and introduces significant latency, which hinders real-time application. Furthermore, traditional sequence models like Long Short-Term Memory (LSTM) networks often struggle to capture the long-range dependencies inherent in complex surgical procedures. This research explores the use of Transformer models to perform recognition and prediction using exclusively kinematic data, streamlining the computational requirements while maintaining high performance.
Methodology
The authors adapted the standard Transformer encoder-decoder architecture, originally designed for natural language translation, to handle continuous surgical data. The model was tasked with three distinct objectives:
- Gesture Recognition: Identifying the current surgical action (e.g., "positioning needle") based on a window of recent robot movements.
- Gesture Prediction: Anticipating the next discrete gesture in the sequence before it occurs.
- Trajectory Prediction: Forecasting the physical path ($x, y, z$ coordinates) of the robot's end-effectors over a short time horizon.
Architectural Adaptations
To transition the Transformer from NLP to robotics, several modifications were implemented:
- Removal of Embedding Layers: Unlike NLP, where discrete words are mapped to vectors, the input here consists of continuous numerical kinematic data. Consequently, standard embedding layers were removed.
- Elimination of Padding Masks: The researchers utilized fixed-size observation windows, rendering the padding masks used for variable-length sentences in NLP unnecessary.
- Encoder-Decoder Mapping: A fully connected layer was added to the encoder to map its output dimensions to the requirements of the decoder, facilitating the attention mechanism between the two components.
By utilizing the "Attention" mechanism, the model can weigh the importance of different time steps within an observation window, allowing it to focus on the specific movements that define a gesture transition.
Experimental Setup
The models were evaluated using the JIGSAWS (JHU-ISI Gesture and Skill Assessment Working Set) dataset, specifically focusing on the "Suturing" task.
- Data Input: The model processed 38 kinematic features per frame, including end-effector positions, rotation matrices, linear and angular velocities, and gripper angles.
- Ground Truth: The dataset includes 16 distinct gesture classes, such as G1 (Reaching for needle) and G3 (Pushing needle through tissue).
- Validation: To ensure the model's ability to generalize across different surgeons, the team used Leave-One-User-Out (LOUO) cross-validation.
Key Results and Discussion
The experimental results demonstrate that kinematic data contains a sufficiently rich "signature" of surgical intent to allow Transformers to outperform multi-modal models.
- Gesture Recognition: The model achieved 89.3% accuracy, surpassing the "Fusion-KVE" method (86.3%) which requires both video and kinematic data.
- Gesture Prediction: The model predicted gestures 1 second into the future with 84.6% accuracy.
- Trajectory Prediction: For a 1-second horizon, the model achieved a Mean Absolute Error (MAE) of only 2.71mm.
These findings suggest that the Transformer's ability to view an entire observation window simultaneously—rather than step-by-step like an RNN—provides a superior global understanding of surgical sub-tasks. The study effectively treats gesture recognition as a "translation" problem, successfully converting physical movement sequences into semantic labels.
Limitations and Future Work
While the results are promising, the current evaluation tested prediction models using "ground truth" inputs for preceding steps. In a deployed autonomous pipeline, a prediction model would rely on the potentially imperfect output of a recognition model. Future research will focus on end-to-end integration to test how error propagation affects the system's robustness. Additionally, further ablation studies are needed to determine why certain gestures remain more difficult to predict than others, paving the way for more reliable real-time surgical assistants.
Summary updated