SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of Surgical Tools

Luca Sestini, Benoit Rosa, Elena De Momi, Giancarlo Ferrigno, Nicolas PadoyMedical Image Analysis 2023 · OpenReview

TL;DR

This paper introduces SAF-IS, a framework that significantly reduces the annotation burden for surgical tool instance segmentation by eliminating the need for pixel-level spatial annotations. It achieves this by leveraging readily available binary tool masks and weak binary tool presence labels, demonstrating competitive performance against fully-supervised methods.

What Problem Does This Paper Address?

Instance segmentation of surgical tools is crucial for many computer-assisted surgery applications, such as skill assessment and augmented reality. However, existing methods rely on fully-supervised deep learning models that require expensive and time-consuming pixel-level annotations. This limits their scalability and generalization. The authors aim to overcome this by developing a framework that can be trained with significantly less, and cheaper, forms of supervision.

Methods

The SAF-IS framework comprises three core components:

  • Tool Instantiation: This module learns to extract individual tool instance masks from binary segmentation masks. It uses a pseudo-supervision signal derived from connected components of the binary masks and a novel "instance pasting augmentation" to simulate overlapping tools, addressing a key challenge in unsupervised instantiation.
  • Feature Representation Learning: A self-supervised approach using contrastive learning is employed to encode each tool instance into a compact feature vector that captures its semantic content. This is achieved by leveraging intrinsic temporal information from video sequences and local instance tracking to generate positive and negative samples.
  • Instance Classification: This module learns to classify tool instances using binary tool presence labels (either frame-wise or sequence-wise), which are much cheaper to obtain than spatial annotations. It uses a teacher-student approach where a teacher network, trained on a small set of human-labeled "prototype instances" (selected via unsupervised clustering), guides the training of a student network on the weak labels.

Datasets

The framework is validated on the following datasets:

  • MICCAI 2017 EndoVis Robotic Instrument Segmentation Challenge dataset
  • MICCAI 2018 EndoVis Robotic Instrument Segmentation Challenge dataset

Key Insights and Ideas

  • Weak Supervision is Powerful: The paper demonstrates that readily available, but spatially unlocalized, binary tool presence labels (frame-wise and sequence-wise) can be effectively used for instance classification, significantly reducing annotation costs.
  • Unsupervised Instantiation is Feasible: By combining connected components analysis with a novel instance pasting augmentation, the authors show that it's possible to generate accurate instance masks without ground truth spatial annotations.
  • Prototype Instances are Key: A small number of carefully selected "prototype instances" (as few as 8) are sufficient to provide effective supervision for the instance classification stage, highlighting the efficiency of targeted human annotation.
  • Sequence-wise Labels are Underrated: The framework effectively utilizes sequence-wise binary tool presence labels, which are even cheaper to obtain than frame-wise labels, showing minimal performance degradation.

Key Results

  • Tool Instantiation: The proposed SAF-IS approach outperforms Mask-RCNN in tool instantiation quality across both datasets and various training modalities, especially when using unsupervised binary masks. For example, on EndoVis 2017, SAF-IS achieved a +417.5% improvement in [email protected] when trained with FUN-SIS predicted masks compared to Mask-RCNN.
  • Instance Segmentation: SAF-IS, trained with only prototype labels and weak binary tool presence labels, outperforms several state-of-the-art fully-supervised and semi-supervised semantic segmentation methods on the EndoVis datasets. For instance, it achieves competitive results even when compared to methods that utilize temporal information and additional tool-part annotations.
  • Annotation Efficiency: The ablation study shows that reducing the number of prototype instances to 8 (approximately 0.26% of training data) results in only a marginal performance drop, underscoring the framework's efficiency.

Limitations and Future Work

  • Feature Learning Bottleneck: The paper identifies feature learning as a crucial methodological bottleneck. While currently unsupervised, incorporating weak information during feature learning could further improve performance.
  • Temporal Modeling: The authors suggest that temporal modeling, currently used for feature learning, could be directly integrated into the inference stage to further enhance segmentation.
  • Leveraging New Segmentation Models: The authors propose that future work could leverage recent advancements like SAM (Segment Anything Model) to obtain initial instance masks, which SAF-IS could then refine and group.
  • Directly Using Weak Labels for Feature Learning: The paper suggests that weak labels could be incorporated earlier in the pipeline, specifically during the feature learning stage, to guide positive and negative sample selection.

Summary updated

Abstract

Instance segmentation of surgical instruments is a long-standing research problem, crucial for the development of many applications for computer-assisted surgery. This problem is commonly tackled via fully-supervised training of deep learning models, requiring expensive pixel-level annotations to train. In this work, we develop a framework for instance segmentation not relying on spatial annotations for training. Instead, our solution only requires binary tool masks, obtainable using recent unsupervised approaches, and binary tool presence labels, freely obtainable in robot-assisted surgery. Based on the binary mask information, our solution learns to extract individual tool instances from single frames, and to encode each instance into a compact vector representation, capturing its semantic features. Such representations guide the automatic selection of a tiny number of instances (8 only in our experiments), displayed to a human operator for tool-type labelling. The gathered information is finally used to match each training instance with a binary tool presence label, providing an effective supervision signal to train a tool instance classifier. We validate our framework on the EndoVis 2017 and 2018 segmentation datasets. We provide results using binary masks obtained either by manual annotation or as predictions of an unsupervised binary segmentation model. The latter solution yields an instance segmentation approach completely free from spatial annotations, outperforming several state-of-the-art fully-supervised segmentation approaches.
BibTeX
@article{sestini2023saf,
  author = {Luca Sestini and Benoit Rosa and Elena De Momi and Giancarlo Ferrigno and Nicolas Padoy},
  title = {{SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of Surgical Tools}},
  journal = {Medical Image Analysis},
  year = {2023},
  url = {https://openreview.net/forum?id=aD9DsUp9ip},
}
Figure