Image Compositing for Segmentation of Surgical Tools Without Manual Annotations

Luis C. García-Peraza-Herrera, Lucas Fidon, Claudia D'Ettorre, Danail Stoyanov, Tom Vercauteren, Sébastien Ourselin

TL;DR

This paper tackles the challenge of segmenting surgical tools in real-time by proposing a novel method to automatically generate realistic training data. By compositing foreground instrument images (captured with a green screen) onto background surgical scenes, they create a large, semi-synthetic dataset that allows training a segmentation model to perform on par with models trained on manually annotated real data, significantly reducing annotation costs.

What Problem Does This Paper Address?

Training deep learning models for tasks like surgical tool segmentation requires vast amounts of precisely labeled data. Manually annotating surgical images, pixel by pixel, is incredibly time-consuming, expensive, and often a bottleneck in developing robust computer-assisted interventions. Existing datasets are often too small to train state-of-the-art models effectively. The authors aim to bridge this gap by creating a method to automatically generate a large, realistic training dataset for surgical tool segmentation, thereby circumventing the need for extensive manual annotation.

Methods

The core of the paper lies in their semi-synthetic data generation approach. They capture foregrounds (surgical instruments) against a chroma key (green screen) and backgrounds (surgical scenes without instruments). The key innovation is how they blend these two components to create training images.

  • Chroma Key Compositing: Instruments are easily extracted from their green screen background.
  • Blending Functions: They explore various blending techniques (trivial, Gaussian, Laplacian pyramid) to combine foregrounds and backgrounds.
  • Mix-Blend Learning: A novel approach where multiple blending functions are probabilistically combined to create a single training image. This encourages the model to be invariant to the specific blending method used, thus improving generalization.
  • Data Augmentation: Extensive augmentation is applied to both foregrounds and backgrounds (zooming, flipping, brightness changes, etc.) and to the final blended images (synthetic smoke, shadows, blood droplets) to increase robustness.
  • Post-processing: GrabCut, a semi-automatic segmentation technique, is used to refine the model's output on real data.

Datasets

  • Foreground Dataset: 14K images of surgical instruments captured over a chroma key.
  • Background Dataset: 6K images from 50 surgical procedures, containing only tissue.
  • Semi-synthetic Training Dataset: 100K labeled images generated by compositing foregrounds and backgrounds.
  • Real Clinical Testing Sets:
    • EndoVis2017: A publicly available dataset with manually labeled surgical instrument segmentation.
  • RoboTool: A newly created dataset by the authors, containing 514 manually annotated images from 20 surgical procedures.

Key Insights and Ideas

  • Chroma Key as a Powerful Tool: The simple chroma key technique, borrowed from special effects, is surprisingly effective for isolating surgical instruments.
  • Blending Matters for Generalization: The way instruments are blended onto backgrounds significantly impacts the model's ability to generalize to real-world data. Simple blending methods can introduce artifacts that the model might learn, hindering performance.
  • Mix-Blend Outperforms Simpler Methods: The proposed "mix-blend" approach, which probabilistically combines multiple blending functions, leads to better segmentation accuracy than using individual blending methods or a fixed combination (multi-blend). This suggests that exploring a wider variety of blending is crucial.
  • Semi-Synthetic Data Can Match Real Data Performance: With the right blending strategy and post-processing, models trained solely on semi-synthetic data can achieve performance comparable to models trained on manually annotated real datasets.
  • Post-processing Bridges the Gap: Even with semi-synthetic training, a simple post-processing step like GrabCut can further improve performance on real data.

Key Results

  • A U-Net trained solely on their semi-synthetic data with the "mix-blend" approach and GrabCut post-processing achieved performance comparable to a U-Net trained on the publicly available EndoVis2017 dataset.
  • Specifically, on the RoboTool dataset, their mix-blend approach with GrabCut post-processing achieved an IoU of 68.1% [42.6, 92.5], which is close to the performance of a model trained on the EndoVis2017 dataset (69.4% [37.7, 91.8] with GrabCut post-processing).
  • The "mix-blend" method outperformed other blending strategies, including "multi-blend" (a fixed combination of blending functions), by a significant margin (e.g., 8 percentage points on EndoVis2017).

Limitations and Future Work

  • Domain Gap: While their method significantly reduces the domain gap, it's acknowledged that a gap can still exist between the semi-synthetic data and real clinical videos, especially concerning specific tool-tissue interactions.
  • Flexibility vs. Realism: Recording foregrounds with the exact target imaging system (e.g., endoscope) could further reduce the domain gap but would sacrifice the method's flexibility.
  • Future Work: The authors suggest exploring the generalization performance when recording foregrounds with different endoscopes. They also mention that incorporating a small amount of manually annotated images displaying specific tool-tissue interactions that are hard to synthesize could further improve performance. They also propose exploring other domain adaptation techniques.

Summary updated

Abstract

Producing manual, pixel-accurate, image segmentation labels is tedious and time-consuming. This is often a rate-limiting factor when large amounts of labeled images are required, such as for training deep convolutional networks for instrument-background segmentation in surgical scenes. No large datasets comparable to industry standards in the computer vision community are available for this task. To circumvent this problem, we propose to automate the creation of a realistic training dataset by exploiting techniques stemming from special effects and harnessing them to target training performance rather than visual appeal. Foreground data is captured by placing sample surgical instruments over a chroma key (a.k.a. green screen) in a controlled environment, thereby making extraction of the relevant image segment straightforward. Multiple lighting conditions and viewpoints can be captured and introduced in the simulation by moving the instruments and camera and modulating the light source. Background data is captured by collecting videos that do not contain instruments. In the absence of pre-existing instrument-free background videos, minimal labeling effort is required, just to select frames that do not contain surgical instruments from videos of surgical interventions freely available online. We compare different methods to blend instruments over tissue and propose a novel data augmentation approach that takes advantage of the plurality of options. We show that by training a vanilla U-Net on semi-synthetic data only and applying a simple post-processing, we are able to match the results of the same network trained on a publicly available manually labeled real dataset.
BibTeX
@article{garciaperazaherrera2021image,
  author = {Luis C. García-Peraza-Herrera and Lucas Fidon and Claudia D'Ettorre and Danail Stoyanov and Tom Vercauteren and Sébastien Ourselin},
  title = {{Image Compositing for Segmentation of Surgical Tools Without Manual Annotations}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2021},
  url = {https://openreview.net/forum?id=mI9EgJMIEe},
}
Figure