Real-Time Segmentation of Non-rigid Surgical Tools Based on Deep Learning and Tracking

Luis C. García-Peraza-Herrera, Wenqi Li, Caspar Gruijthuijsen, Alain Devreker, George Attilakos, Jan Deprest, Emmanuel B. Vander Poorten, Danail Stoyanov, Tom Vercauteren, Sébastien Ourselin

TL;DR

This paper introduces a novel real-time system for segmenting surgical tools in endoscopic videos by combining deep learning (Fully Convolutional Networks - FCNs) with optical flow tracking. This approach achieves state-of-the-art accuracy in non-real-time segmentation and provides a practical real-time solution, crucial for enhancing surgeon awareness and enabling advanced computer-assisted surgery applications.

What Problem Does This Paper Address?

Computer-assisted surgery (CAS) systems rely heavily on understanding the surgical environment. A key component of this is accurately identifying and segmenting surgical tools. This is particularly challenging due to the deformable nature of some instruments, varying lighting conditions, specular reflections, and occlusions common in endoscopic views. Existing methods often struggle to provide real-time, accurate segmentations, especially for non-rigid tools, limiting their integration into live surgical workflows. The paper aims to bridge this gap by developing a robust and efficient method for real-time surgical tool segmentation.

Methods

The core of the proposed method is a two-pronged approach:

  1. Deep Learning Segmentation (FCN): The authors leverage Fully Convolutional Networks (FCNs), specifically an adapted FCN-8s architecture. FCNs are chosen for their ability to perform semantic segmentation (pixel-level classification) and their capacity to learn features automatically, eliminating the need for manual feature engineering. A key advantage is that pre-trained FCNs can be fine-tuned on relatively small medical datasets, addressing the scarcity of labeled surgical data. The FCN is adapted to output only two scores per pixel: background and foreground.
  2. Optical Flow Tracking: To achieve real-time performance, the system employs an asynchronous pipeline. When the FCN is not busy, it processes a frame to generate a segmentation mask. In the interim frames, optical flow (specifically, pyramidal Lucas-Kanade with RANSAC for affine transformation estimation) is used to propagate the last FCN-generated segmentation mask. This "mask propagation" approach avoids the computational cost of running the FCN on every single frame, enabling real-time speeds. The method is designed to be instrument-agnostic.

Datasets

The methodology was validated using three datasets:

  • EndoVisSub (robotic and non-robotic): A benchmark dataset from the MICCAI 2015 Endoscopic Vision Challenge, featuring ex vivo robotic surgery and in vivo abdominal surgery.
  • NeuroSurgicalTools: A dataset of in vivo neurosurgeries, known for its challenging conditions.
  • FetalFlexTool: A newly introduced dataset featuring ex vivo fetal surgery with a non-rigid McKibben artificial muscle, captured under varying conditions (air and water) to test generalization.

Key Insights and Ideas

  • FCNs for Surgical Tool Segmentation: Deep learning, particularly FCNs, offers a powerful way to segment surgical tools without manual feature engineering, adapting well to the complex visual characteristics of surgical environments.
  • Fine-tuning is Key: The ability to fine-tune pre-trained FCNs on small, domain-specific datasets is crucial for overcoming the limited availability of labeled surgical data.
  • Optical Flow for Real-Time Performance: Combining a powerful but computationally intensive deep learning model with a faster optical flow tracking mechanism is an effective strategy for achieving real-time segmentation. This hybrid approach leverages the strengths of both techniques.
  • Asynchronous Pipeline: Running the FCN segmenter asynchronously to the video feed and using mask propagation for intermediate frames is essential for meeting real-time constraints.
  • Generalization Across Instrument Types: The method demonstrates flexibility by successfully segmenting rigid, articulated, and flexible instruments across different surgical scenarios.

Key Results

  • Non-Real-Time (FCN only): Achieved a balanced accuracy of 89.6% on the NeuroSurgicalTools dataset, outperforming the state-of-the-art by 3.8 percentage points. The FCN-based segmentation achieved high balanced accuracies across all tested datasets (e.g., 83.7% for EndoVisSub robotic, 89.6% for NeuroSurgicalTools, 92.3% for FetalFlexTool).
  • Real-Time (FCN + Optical Flow): The combined pipeline achieved an average balanced accuracy of 78.2% across all validated datasets. The real-time pipeline was able to produce results at approximately 30Hz for all datasets.
  • Qualitative results showed the system's ability to handle challenges like specular reflections, shadows, and blurriness.

Limitations and Future Work

  • Non-Rigid Tool Movement: The paper acknowledges that non-rigid foreground movements faster than the FCN's processing time (typically 100ms) can affect segmentation quality and may not be captured as well.
  • Specificity Reduction in Real-Time: The real-time version, while achieving good overall accuracy, sometimes shows a reduction in specificity compared to the non-real-time FCN-only approach, potentially due to the propagation of warped segmentations.
  • Future Work:
    • Investigating the detection and segmentation of multiple instruments simultaneously.
  • Integrating long-term tracking information to further improve segmentation results and potentially speed up the process.
  • Exploring more sophisticated methods for handling non-rigid foreground movements, such as grouping feature points and using multiple affine transformations.

Summary updated

Abstract

Real-time tool segmentation is an essential component in computer-assisted surgical systems. We propose a novel real-time automatic method based on Fully Convolutional Networks (FCN) and optical flow tracking. Our method exploits the ability of deep neural networks to produce accurate segmentations of highly deformable parts along with the high speed of optical flow. Furthermore, the pre-trained FCN can be fine-tuned on a small amount of medical images without the need to hand-craft features. We validated our method using existing and new benchmark datasets, covering both ex vivo and in vivo real clinical cases where different surgical instruments are employed. Two versions of the method are presented, non-real-time and real-time. The former, using only deep learning, achieves a balanced accuracy of 89.6% on a real clinical dataset, outperforming the (non-real-time) state of the art by 3.8% points. The latter, a combination of deep learning with optical flow tracking, yields an average balanced accuracy of 78.2% across all the validated datasets.
BibTeX
@inproceedings{garciaperazaherrera2016real,
  author = {Luis C. García-Peraza-Herrera and Wenqi Li and Caspar Gruijthuijsen and Alain Devreker and George Attilakos and Jan Deprest and Emmanuel B. Vander Poorten and Danail Stoyanov and Tom Vercauteren and Sébastien Ourselin},
  title = {{Real-Time Segmentation of Non-rigid Surgical Tools Based on Deep Learning and Tracking}},
  booktitle = {CARE @ MICCAI},
  year = {2016},
  url = {https://openreview.net/forum?id=nQ2r9d2qMH},
}
Figure