Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery

Sebastian Bodenstedt, Max Allan, Anthony Agustinos, Xiaofei Du, Luis C. García-Peraza-Herrera, Hannes Kenngott, Thomas Kurmann, Beat P. Müller-Stich, Sébastien Ourselin, Daniil Pakhomov, Raphael Sznitman, Marvin Teichmann, Martin Thoma, Tom Vercauteren, Sandrine Voros, Martin Wagner, Pamela Wochner, Lena Maier-Hein, Danail Stoyanov, Stefanie SpeidelarXiv 2018 · OpenReview

Abstract

Intraoperative segmentation and tracking of minimally invasive instruments is a prerequisite for computer- and robotic-assisted surgery. This paper presents a comparative validation study of different vision-based methods for instrument segmentation and tracking in the context of robotic and conventional laparoscopic surgery. We introduce a comprehensive validation dataset and present the results of the comparative study. Findings indicate that modern deep learning approaches outperform other methods in segmentation, and merging results from multiple methods significantly increases accuracy. However, instrument tracking remains an open challenge, particularly in conventional laparoscopic scenarios.

Introduction

In minimally invasive surgery (MIS), precise localization of instruments is essential for providing surgeons with augmented reality overlays—such as "seeing through" tissue to find hidden vessels—and for automating robotic tasks. While hardware-based tracking (using sensors on the tools) exists, it is often cumbersome, expensive, and prone to inaccuracies.

Computer vision offers a "hardware-free" alternative by analyzing endoscopic video directly. However, the field has been hindered by a lack of standardized, publicly available datasets. This study addresses this gap by organizing the Endoscopic Vision Challenge, providing a common yardstick to evaluate algorithms in the messy, unpredictable environment of a real operating room.

Datasets and Methodology

The authors introduced two key datasets for benchmarking:

  1. D-ROB (Robotic): Six videos of a da Vinci robot in an ex-vivo setting. These provide high accuracy because ground truth labels were generated using the robot's internal kinematics.
  2. D-CONV (Conventional): Six videos from real in-vivo laparoscopic colorectal surgeries. These are significantly more challenging, featuring blood, smoke, and complex tissue backgrounds.
Original endoscopic image showing a surgical instrument in a complex tissue environment.

Participants tested a variety of algorithms. Segmentation methods included traditional Random Forests using hand-crafted color and gradient features, as well as modern Fully Convolutional Networks (FCNs) like VGG-16. Tracking approaches included optical flow, 3D model alignment, and online learning classifiers trained on the fly.

Results and Discussion

The Superiority of Deep Learning and Ensembles

Deep learning models demonstrated superior performance in segmentation tasks. Unlike traditional machine learning that might be confused by a pixel's color, CNNs recognize the overall shape and structure of a surgical tool by leveraging global context.

Visualization of instrument segmentation where the tool is identified with color-coded masks.

A striking finding was the "wisdom of the crowd" effect. Merging the outputs of different models (using techniques like Majority Voting or the STAPLE algorithm) yielded better results than any single standalone model. The best individual models achieved a Dice Similarity Coefficient (DSC) of approximately 0.88, which rose to 0.89 when merged into an ensemble. This suggests that different models fail in different ways, and their collective consensus filters out individual errors.

Challenges in Tracking

While segmentation is reaching high levels of accuracy, tracking remains the "final frontier." Fast movements, occlusions by tissue, and visual interference like smoke or lens fogging frequently cause even the best trackers to fail. The study showed that methods working well on robotic data (D-ROB) did not always translate to conventional surgery (D-CONV), highlighting a significant dataset bias and the need for more robust, real-world training data.

Conclusion and Future Work

This comparative study establishes a vital benchmark for surgical instrument vision. While deep learning and ensemble methods have advanced segmentation, robust tracking in conventional laparoscopic conditions remains an open challenge. Future work must focus on real-time constraints—as these models need to run at 30+ frames per second for clinical use—and the deeper incorporation of temporal video continuity into neural network architectures.

Summary updated

Abstract

Intraoperative segmentation and tracking of minimally invasive instruments is a prerequisite for computer- and robotic-assisted surgery. Since additional hardware like tracking systems or the robot encoders are cumbersome and lack accuracy, surgical vision is evolving as promising techniques to segment and track the instruments using only the endoscopic images. However, what is missing so far are common image data sets for consistent evaluation and benchmarking of algorithms against each other. The paper presents a comparative validation study of different vision-based methods for instrument segmentation and tracking in the context of robotic as well as conventional laparoscopic surgery. The contribution of the paper is twofold: we introduce a comprehensive validation data set that was provided to the study participants and present the results of the comparative validation study. Based on the results of the validation study, we arrive at the conclusion that modern deep learning approaches outperform other methods in instrument segmentation tasks, but the results are still not perfect. Furthermore, we show that merging results from different methods actually significantly increases accuracy in comparison to the best stand-alone method. On the other hand, the results of the instrument tracking task show that this is still an open challenge, especially during challenging scenarios in conventional laparoscopic surgery.
BibTeX
@misc{bodenstedt2018comparative,
  author = {Sebastian Bodenstedt and Max Allan and Anthony Agustinos and Xiaofei Du and Luis C. García-Peraza-Herrera and Hannes Kenngott and Thomas Kurmann and Beat P. Müller-Stich and Sébastien Ourselin and Daniil Pakhomov and Raphael Sznitman and Marvin Teichmann and Martin Thoma and Tom Vercauteren and Sandrine Voros and Martin Wagner and Pamela Wochner and Lena Maier-Hein and Danail Stoyanov and Stefanie Speidel},
  title = {{Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery}},
  year = {2018},
  eprint = {1805.02475},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.02475},
}
Figure