Self-Supervised Surgical Tool Segmentation using Kinematic Information

Cristian da Costa Rocha, Nicolas Padoy, Benoit Rosa

TL;DR

This paper introduces a self-supervised method for segmenting surgical tools in endoscopic video by leveraging the robot's own kinematic data. By using an optimization strategy to align robot models with images, the system generates its own training labels, achieving performance nearly on par with fully supervised models without requiring a single manually annotated image.

What Problem Does This Paper Address?

In robot-assisted surgery, knowing exactly where a tool is in a camera's field of view (segmentation) is the first step toward automation and advanced safety features. Currently, the "gold standard" involves training Deep Learning models on thousands of manually labeled images.

However, manual annotation is a massive bottleneck: it is expensive, time-consuming, and often doesn't generalize well because every surgical setup and patient is slightly different. While robots "know" where their arms are via kinematics (joint angles), this data is often too imprecise for direct vision tasks due to cable friction, tissue interaction, and unknown camera-to-robot calibration (the "hand-eye" problem).

Methods

The authors propose a two-step "SSTS" (Self-Supervised Tool Segmentation) workflow:

  1. Kinematic Optimization: At the start of a procedure, the robot moves through a few poses. The system projects a 3D model of the robot onto the 2D image based on initial (imprecise) kinematics. It then uses a Grabcut-based cost function to iteratively refine the hand-eye calibration. The intuition is that if the projection is accurate, it should align well with the actual tool boundaries found by the Grabcut algorithm.
  2. FCN Training: Once the calibration is optimized, the system generates "self-labeled" masks for those initial frames. These masks are used to fine-tune a Fully Convolutional Network (FCN) based on a ResNet18 architecture.

By training the FCN on these self-generated labels, the system learns to recognize the tool's visual appearance. This allows it to segment the tool throughout the rest of the surgery, even when the kinematic model becomes inaccurate (e.g., when the tool bends while pushing against tissue).

Datasets

The approach was validated on three distinct datasets using the STRAS (a flexible robotized endoscopy system) robot:

  • Phantom 1: A plastic model of the human digestive system.
  • Phantom 2: A silicone model of the human colon.
  • In Vivo: Data from a porcine surgery, featuring challenging conditions like smoke, blood, and tissue interaction.
  • Ground Truth: For validation only, 30 images from each dataset were manually segmented to compare the self-supervised results against human performance.

Key Insights and Ideas

  • Kinematics as a "Noisy Teacher": The core "aha" moment is that while raw kinematics are too noisy to use directly for segmentation, they provide enough of a "hint" to initialize an optimization process that generates high-quality labels.
  • Closing the Loop with Grabcut: Using Grabcut (a classic computer vision segmentation tool) as part of the cost function allows the system to bridge the gap between abstract robot coordinates and actual image pixels.
  • Generalization through Deep Learning: The FCN isn't just a copy of the kinematics; it learns the visual features of the tool. This makes the system robust to the very errors (like mechanical flex) that make raw kinematics unreliable.

Key Results

The SSTS approach performed remarkably well compared to Fully Supervised Learning (FSL), which requires manual labels:

  • Accuracy: On the Phantom 1 dataset, SSTS achieved an IoU (Intersection over Union) of 0.86, nearly matching the FSL score of 0.87.
  • In Vivo Performance: In the much tougher porcine environment, SSTS reached an IoU of 0.62 (vs. 0.72 for FSL). While lower, this is a significant result for a system that received zero human guidance.
  • Robustness: The FCN training step proved essential. Using the optimized kinematic projection alone (Grabcut only) yielded much lower scores (e.g., 0.56 IoU on Phantom 1), proving that the neural network successfully "learned" to overcome remaining kinematic inaccuracies.

Limitations and Future Work

  • Processing Speed: The current implementation takes about 43 minutes (900s for optimization + 1700s for training), which is too slow for immediate "plug-and-play" use in a live OR. The authors suggest GPU-accelerated Grabcut could provide a 10x speedup.
  • Bimanual Complexity: The study focused on a single robotic arm. In the in vivo test, a second arm was present; the system occasionally confused parts of the second arm for the first due to their similar appearance.
  • Environmental Changes: The authors plan to investigate running the SSTS process at regular intervals during surgery to adapt to changing lighting or environments (e.g., the lens getting dirty or significant changes in background tissue).

Summary updated

Abstract

Surgical tool segmentation in endoscopic images is the first step towards pose estimation and (sub-)task automation in challenging minimally invasive surgical operations. While many approaches in the literature have shown great results using modern machine learning methods such as convolutional neural networks, the main bottleneck lies in the acquisition of a large number of manually-annotated images for efficient learning. This is especially true in surgical context, where patient-to-patient differences impede the overall generalizability. In order to cope with this lack of annotated data, we propose a self-supervised approach in a robot-assisted context. To our knowledge, the proposed approach is the first to make use of the kinematic model of the robot in order to generate training labels. The core contribution of the paper is to propose an optimization method to obtain good labels for training despite an unknown hand-eye calibration and an imprecise kinematic model. The labels can subsequently be used for fine-tuning a fully-convolutional neural network for pixel-wise classification. As a result, the tool can be segmented in the endoscopic images without needing a single manually-annotated image. Experimental results on phantom and in vivo datasets obtained using a flexible robotized endoscopy system are very promising.
BibTeX
@inproceedings{rocha2019self,
  author = {Cristian da Costa Rocha and Nicolas Padoy and Benoit Rosa},
  title = {{Self-Supervised Surgical Tool Segmentation using Kinematic Information}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=wyUqWM5sRl},
}

Loading related content...

Figure