Self-Supervised Surgical Tool Segmentation using Kinematic Information

Cristian da Costa Rocha, Nicolas Padoy, Benoit Rosa

Abstract

Surgical tool segmentation in endoscopic images is the first step towards pose estimation and (sub-)task automation in challenging minimally invasive surgical operations. While many approaches in the literature have shown great results using modern machine learning methods such as convolutional neural networks, the main bottleneck lies in the acquisition of a large number of manually-annotated images for efficient learning. This is especially true in surgical context, where patient-to-patient differences impede the overall generalizability. In order to cope with this lack of annotated data, we propose a self-supervised approach in a robot-assisted context. To our knowledge, the proposed approach is the first to make use of the kinematic model of the robot in order to generate training labels. The core contribution of the paper is to propose an optimization method to obtain good labels for training despite an unknown hand-eye calibration and an imprecise kinematic model. The labels can subsequently be used for fine-tuning a fully-convolutional neural network for pixel-wise classification. As a result, the tool can be segmented in the endoscopic images without needing a single manually-annotated image. Experimental results on phantom and in vivo datasets obtained using a flexible robotized endoscopy system are very promising.

Introduction

In robot-assisted surgery, precise tool segmentation is a prerequisite for automation and enhanced safety. Current state-of-the-art methods rely heavily on supervised deep learning, which requires thousands of manually annotated images. This annotation process is a significant bottleneck—it is labor-intensive, costly, and often fails to generalize across different surgical setups or patients.

While surgical robots provide kinematic data (joint angles), this information is typically too imprecise for direct vision tasks due to mechanical factors like cable friction and unknown camera-to-robot (hand-eye) calibration. This paper introduces a self-supervised method that leverages these "noisy" kinematics to generate training labels automatically, eliminating the need for manual annotation.

Methodology

The proposed Self-Supervised Tool Segmentation (SSTS) workflow consists of two primary stages: kinematic optimization and neural network training.

Overview of the self-supervised tool segmentation (SSTS) pipeline.

Kinematic Optimization

At the beginning of a procedure, the robot performs a series of movements. A 3D model of the robot is projected onto the 2D endoscopic image based on initial, imprecise kinematics. An optimization process then refines the hand-eye calibration by minimizing a Grabcut-based cost function. This function aligns the projected model with the tool's visual boundaries in the image.

FCN Training

The optimized projections serve as "self-generated" labels for the initial frames. These labels are used to fine-tune a Fully Convolutional Network (FCN) based on a ResNet18 architecture. By training on these labels, the FCN learns the visual features of the tool, allowing it to maintain accurate segmentation even when kinematic models drift during the surgery (e.g., due to tool-tissue interaction).

Experimental Results

The SSTS approach was evaluated on three datasets using the STRAS flexible endoscopy system: two phantom models (digestive system and colon) and an in vivo porcine surgery dataset.

Qualitative results showing the original image, generated masks, and final segmentation.

The results demonstrate that SSTS performs comparably to Fully Supervised Learning (FSL):

  • Accuracy: On the Phantom 1 dataset, SSTS achieved an Intersection over Union (IoU) of 0.86, nearly matching the FSL score of 0.87.
  • In Vivo Performance: In challenging porcine data featuring smoke and blood, SSTS reached an IoU of 0.62 compared to 0.72 for FSL.
  • Robustness: The FCN training step was critical; using optimized kinematic projections alone yielded significantly lower performance (e.g., 0.56 IoU on Phantom 1), proving the network's ability to generalize beyond kinematic errors.

Discussion and Future Work

The core innovation of this work is treating kinematics as a "noisy teacher" to initialize a visual learning process. This bridges the gap between abstract robot coordinates and pixel-level image data.

However, challenges remain. The current pipeline takes approximately 43 minutes to optimize and train, which is not yet suitable for real-time clinical deployment. Future work will focus on GPU acceleration for the Grabcut component. Additionally, the system occasionally struggled with bimanual tasks where tools had similar visual appearances. Future iterations will explore periodic re-training to adapt to changing surgical environments, such as lens contamination or varying lighting conditions.

Summary updated

Abstract

Surgical tool segmentation in endoscopic images is the first step towards pose estimation and (sub-)task automation in challenging minimally invasive surgical operations. While many approaches in the literature have shown great results using modern machine learning methods such as convolutional neural networks, the main bottleneck lies in the acquisition of a large number of manually-annotated images for efficient learning. This is especially true in surgical context, where patient-to-patient differences impede the overall generalizability. In order to cope with this lack of annotated data, we propose a self-supervised approach in a robot-assisted context. To our knowledge, the proposed approach is the first to make use of the kinematic model of the robot in order to generate training labels. The core contribution of the paper is to propose an optimization method to obtain good labels for training despite an unknown hand-eye calibration and an imprecise kinematic model. The labels can subsequently be used for fine-tuning a fully-convolutional neural network for pixel-wise classification. As a result, the tool can be segmented in the endoscopic images without needing a single manually-annotated image. Experimental results on phantom and in vivo datasets obtained using a flexible robotized endoscopy system are very promising.
BibTeX
@inproceedings{rocha2019self,
  author = {Cristian da Costa Rocha and Nicolas Padoy and Benoit Rosa},
  title = {{Self-Supervised Surgical Tool Segmentation using Kinematic Information}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=wyUqWM5sRl},
}

Loading related content...

Figure