Kinematic Data-Based Action Segmentation for Surgical Applications

Adam Goldbraikh, Omer Shubi, Or Rubin, Carla M. Pugh, Shlomi LauferarXiv 2023 · arXiv · OpenReview

TL;DR

This paper introduces novel deep learning architectures (MS-TCRNet) for action segmentation using kinematic data, achieving state-of-the-art results on surgical tasks. They also propose two new data augmentation techniques, World Frame Rotation and Hand Inversion, to improve model robustness and generalization, particularly for underrepresented scenarios like left-handed surgeons.

What Problem Does This Paper Address?

Action segmentation, the task of labeling each moment in a sequence with a specific action, is crucial for understanding complex processes. While traditionally applied to video, kinematic data from sensors offers advantages like privacy and efficiency by bypassing image feature extraction. However, accurately segmenting actions from kinematic data, especially in dynamic environments like surgery, remains challenging. Existing methods struggle with long-term dependencies and can be sensitive to variations in data collection or surgeon handedness. This paper aims to improve action segmentation accuracy and robustness on kinematic data, with a specific focus on surgical applications.

Methods

The core of this work lies in two main contributions:

  • MS-TCRNet Architectures: The authors propose two multi-stage temporal convolutional recurrent networks (MS-TCRNet), named L-MS-TCRNet (using Bidirectional LSTMs) and G-MS-TCRNet (using Bidirectional GRUs). These networks feature:
  • Intra-Stage Regularization (ISR): A prediction generator with additional prediction heads within its stages. This allows the model to classify actions based on partial information, improving frame-wise performance.
  • BiRNN-based Refinement Stages: Bidirectional LSTM or GRU layers are used for refinement. These stages employ downsampling of the input to reduce noise and optimize frequencies for the RNNs, followed by upsampling to restore the original sequence resolution, which helps reduce over-segmentation errors.
  • Data Augmentation Techniques: To enhance model robustness and generalization, two novel data augmentation methods are introduced:
  • World Frame Rotation (WFR): This technique simulates scenarios where the world coordinate system is rotated. It achieves this by applying a random rotation matrix to the sensor positions and orientations, effectively rotating the entire scene.
  • Hand Inversion (HI): Inspired by horizontal flips in image augmentation, this method simulates a left-handed surgeon's movements by mirroring the data across a plane that separates left and right-hand data points. This is particularly useful for addressing the underrepresentation of left-handed surgeons in datasets.

Datasets

The proposed models and augmentations were evaluated on three datasets:

  • Variable Tissue Simulation (VTS) Dataset: An open surgery simulation dataset collected by the authors, focusing on suturing tasks.
  • Bowel Repair Simulation (BRS) Dataset: A newly introduced open surgery simulation dataset also collected by the authors, featuring enterotomy repair procedures.
  • JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS): A well-known benchmark dataset for robotic surgery, containing tasks like suturing, knot tying, and needle passing.

Key Insights and Ideas

  • Decomposition for Refinement: The multi-stage approach, with a prediction generator followed by RNN-based refinement, effectively separates the tasks of frame-wise prediction and sequence-level segmentation. This allows each stage to specialize, leading to better overall performance.
  • Intra-Stage Regularization for Local Cues: Adding prediction heads within the prediction generator's stages allows the model to leverage local temporal information more effectively, boosting frame-wise accuracy.
  • Downsampling/Upsampling for RNNs: The downsampling and upsampling strategy in the refinement stages is crucial for optimizing RNN performance on kinematic data. It helps manage noise and temporal resolution, leading to more accurate segmentation.
  • Geometric Augmentations are Powerful: Leveraging the inherent geometric structure of kinematic data through WFR and HI proves highly effective. These augmentations go beyond simple transformations and capture realistic variations in data acquisition and user behavior.
  • Addressing Data Imbalance: The Hand Inversion augmentation is a clever way to tackle the problem of underrepresented left-handed surgeons, significantly improving performance for this group without negatively impacting right-handed surgeons.

Key Results

  • State-of-the-Art Performance: The proposed MS-TCRNet architectures (L-MS-TCRNet and G-MS-TCRNet) achieve state-of-the-art results on the VTS and JIGSAWS datasets.
  • BRS Baseline: The paper establishes a strong baseline for the newly introduced BRS dataset, outperforming existing methods like MS-TCN++.
  • Augmentation Benefits: Both WFR and HI augmentations consistently improve performance, particularly on the BRS dataset. WFR shows a noticeable impact, especially when applied with a probability of one. HI significantly boosts performance for left-handed surgeons.
  • Combined Augmentation Power: The combination of WFR and HI yields the best results on the JIGSAWS dataset, demonstrating synergistic benefits.
  • Robustness: Augmentations not only increase mean performance but also reduce standard deviations, indicating improved robustness of the models.

Limitations and Future Work

  • Sensor Dependency: The current method relies on electromagnetic tracking systems for kinematic data. While the authors suggest their architecture can be applied to IMU data, further preprocessing of IMU data (which measures relative to the body frame) might be necessary for optimal performance.
  • Common Frame Assumption: The augmentations assume all sensors are relative to a common world frame. This is true for electromagnetic tracking systems but might require adaptation for other sensor setups.
  • Future Work: The authors aim to address the limitations related to IMU data and explore its direct application. They also plan to investigate the use of raw IMU data for simulating linear and angular velocities in the body frame, enabling performance comparisons with IMU-like data.

Summary updated

Abstract

Action segmentation is a challenging task in high-level process analysis, typically performed on video or kinematic data obtained from various sensors. In the context of surgical procedures, action segmentation is critical for workflow analysis algorithms. This work presents two contributions related to action segmentation on kinematic data. Firstly, we introduce two multi-stage architectures, MS-TCN-BiLSTM and MS-TCN-BiGRU, specifically designed for kinematic data. The architectures consist of a prediction generator with intra-stage regularization and Bidirectional LSTM or GRU-based refinement stages. Secondly, we propose two new data augmentation techniques, World Frame Rotation and Horizontal-Flip, which utilize the strong geometric structure of kinematic data to improve algorithm performance and robustness. We evaluate our models on three datasets of surgical suturing tasks: the Variable Tissue Simulation (VTS) Dataset and the newly introduced Bowel Repair Simulation (BRS) Dataset, both of which are open surgery simulation datasets collected by us, as well as the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), a well-known benchmark in robotic surgery. Our methods achieve state-of-the-art performance on all benchmark datasets and establish a strong baseline for the BRS dataset.
BibTeX
@misc{goldbraikh2023kinematic,
  author = {Adam Goldbraikh and Omer Shubi and Or Rubin and Carla M. Pugh and Shlomi Laufer},
  title = {{Kinematic Data-Based Action Segmentation for Surgical Applications}},
  year = {2023},
  eprint = {2303.07814},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.07814},
}

Loading related content...

Figure