ToolNet: Holistically-nested real-time segmentation of robotic surgical tools

Luis C. García-Peraza-Herrera, Wenqi Li, Lucas Fidon, Caspar Gruijthuijsen, Alain Devreker, George Attilakos, Jan Deprest, Emmanuel B. Vander Poorten, Danail Stoyanov, Tom Vercauteren, Sébastien Ourselin

TL;DR

This paper introduces two novel, lightweight deep learning architectures, ToolNetMS and ToolNetH, for real-time segmentation of robotic surgical tools in endoscopic videos. These networks significantly outperform state-of-the-art methods in accuracy and speed, paving the way for enhanced robotic surgery assistance and data science applications.

What Problem Does This Paper Address?

Minimally invasive robotic surgery offers numerous benefits, but it also introduces challenges. A key issue is the loss of direct tactile feedback, which surgeons rely on for precise manipulation and confirmation of actions. To compensate, researchers are developing computer-assisted systems that can provide visual cues and enhance situational awareness. A crucial component for these systems is the ability to accurately and in real-time segment surgical instruments from endoscopic video feeds. This is a difficult task due to the heterogeneous nature of endoscopic scenes, featuring specular reflections, partial occlusions, blurriness, and complex interactions between instruments and tissue. Existing methods, while improving, often struggle with accuracy, speed, and the inherent class imbalance (few instrument pixels vs. many background pixels).

Methods

The authors propose two novel deep learning architectures, ToolNetMS and ToolNetH, designed for real-time semantic segmentation of robotic surgical tools. Both architectures are significantly smaller than previous state-of-the-art Fully Convolutional Networks (FCNs), leading to faster inference times.

  • ToolNetMS (Multi-Scale): This architecture encodes multi-scale information by cascading predictions from different network layers and then aggregating them. This approach aims to improve segmentation accuracy, particularly at object boundaries.
  • ToolNetH (Holistically-Nested): This architecture takes a different approach to multi-scale integration. Inspired by holistically-nested edge detection, it imposes multi-scale consistency by incorporating the loss at each scale into the overall optimization process. This is achieved through a novel Multi-Scale Dice Loss (MSDL).

Both networks utilize Parametric Rectified Linear Units (PReLU) as activation functions. PReLU is a more robust regularization technique than standard ReLU, learning adaptive slopes for negative inputs, which helps improve segmentation accuracy with minimal computational overhead.

The paper also introduces Dice Loss as the primary objective function. This is particularly well-suited for surgical segmentation tasks where there's a significant imbalance between foreground (instrument) and background pixels, ensuring that the segmentation of smaller foreground objects is not overshadowed.

Datasets

The primary dataset used for training and evaluation is the Da Vinci Robotic (DVR) dataset. This dataset consists of ex vivo robotic surgery videos featuring articulated instruments. The ground truth masks were generated using a combination of joint encoder information and forward kinematics, with manual correction for hand-eye calibration errors. The dataset includes both training and testing videos with varying numbers and types of instruments, including some not present in the training set to test generalization.

Key Insights and Ideas

  • Lightweight Architectures for Real-Time Performance: The core insight is that achieving real-time segmentation for robotic surgery doesn't require massive, computationally expensive networks. By designing smaller, more efficient architectures (ToolNetMS and ToolNetH), the authors demonstrate that high accuracy can be maintained while drastically reducing parameter count and inference time.
  • Multi-Scale Feature Fusion is Crucial: The paper highlights the importance of leveraging features from multiple scales within the network. Both ToolNetMS (via cascaded aggregation) and ToolNetH (via holistically-nested loss) show that effectively combining these multi-scale features leads to more accurate and spatially consistent segmentations.
  • Dice Loss for Class Imbalance: The choice of Dice Loss is a critical factor in handling the inherent class imbalance in surgical segmentation. This loss function directly optimizes for overlap, which is more suitable than cross-entropy for scenarios with vastly different numbers of foreground and background pixels.
  • PReLU for Improved Regularization: The adoption of PReLU as an activation function provides a simple yet effective way to improve network regularization and prevent overfitting, especially in smaller networks trained on limited data.

Key Results

The proposed ToolNet architectures demonstrate significant improvements over the state-of-the-art FCN-8s baseline:

  • Accuracy: ToolNetMS achieved a 1.6% improvement in mean Dice Similarity Coefficient (DSC) and ToolNetH achieved a 3.4% improvement, with both also showing gains in balanced accuracy and mean Intersection over Union (IoU).
  • Speed: The parameter count was reduced by over an order of magnitude (from ~134M for FCN-8s to ~7.3M for ToolNetMS and ~7.4M for ToolNetH). This resulted in a dramatic speedup: ToolNetMS achieved an inference time of 23.2ms (43.1 fps), making it viable for real-time robotic surgery, while ToolNetH achieved 34.6ms (28.9 fps).
  • Qualitative Results: Visual comparisons show that ToolNet architectures produce segmentation masks that more accurately respect the borders of surgical tools and exhibit enhanced spatial consistency compared to FCN-8s.

Limitations and Future Work

  • Generalization to Unseen Instruments: While the paper shows good generalization to instruments not present in the training set, further validation on a wider variety of instruments and surgical scenarios would be beneficial.
  • 3D Localization: The current work focuses on 2D segmentation. Future work could explore extending this to 3D localization of instruments, which would be highly valuable for surgical navigation and planning.
  • Data Augmentation and Transfer Learning: The authors suggest that data augmentation and transfer learning from deeper architectures could further enhance performance.
  • Adversarial Training: Investigating adversarial training is proposed as a potential avenue to increase segmentation accuracy, particularly for robotic surgical instruments.

Summary updated

Abstract

Real-time tool segmentation from endoscopic videos is an essential part of many computer-assisted robotic surgical systems and of critical importance in robotic surgical data science. We propose two novel deep learning architectures for automatic segmentation of non-rigid surgical instruments. Both methods take advantage of automated deep-learning-based multi-scale feature extraction while trying to maintain an accurate segmentation quality at all resolutions. The two proposed methods encode the multi-scale constraint inside the network architecture. The first proposed architecture enforces it by cascaded aggregation of predictions and the second proposed network does it by means of a holistically-nested architecture where the loss at each scale is taken into account for the optimization process. As the proposed methods are for real-time semantic labeling, both present a reduced number of parameters. We propose the use of parametric rectified linear units for semantic labeling in these small architectures to increase the regularization of the network while maintaining the segmentation accuracy. We compare the proposed architectures against state-of-the-art fully convolutional networks. We validate our methods using existing benchmark datasets, including ex vivo cases with phantom tissue and different robotic surgical instruments present in the scene. Our results show a statistically significant improved Dice Similarity Coefficient over previous instrument segmentation methods. We analyze our design choices and discuss the key drivers for improving accuracy.
BibTeX
@inproceedings{garciaperazaherrera2017toolnet,
  author = {Luis C. García-Peraza-Herrera and Wenqi Li and Lucas Fidon and Caspar Gruijthuijsen and Alain Devreker and George Attilakos and Jan Deprest and Emmanuel B. Vander Poorten and Danail Stoyanov and Tom Vercauteren and Sébastien Ourselin},
  title = {{ToolNet: Holistically-nested real-time segmentation of robotic surgical tools}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=gS-4kkwRs8Z},
}
Figure