TL;DR
This paper introduces SAF-IS, a framework that significantly reduces the annotation burden for surgical tool instance segmentation by eliminating the need for pixel-level spatial annotations. It achieves this by leveraging readily available binary tool masks and weak binary tool presence labels, demonstrating competitive performance against fully-supervised methods.
What Problem Does This Paper Address?
Instance segmentation of surgical tools is crucial for many computer-assisted surgery applications, such as skill assessment and augmented reality. However, existing methods rely on fully-supervised deep learning models that require expensive and time-consuming pixel-level annotations. This limits their scalability and generalization. The authors aim to overcome this by developing a framework that can be trained with significantly less, and cheaper, forms of supervision.
Methods
The SAF-IS framework comprises three core components:
- Tool Instantiation: This module learns to extract individual tool instance masks from binary segmentation masks. It uses a pseudo-supervision signal derived from connected components of the binary masks and a novel "instance pasting augmentation" to simulate overlapping tools, addressing a key challenge in unsupervised instantiation.
- Feature Representation Learning: A self-supervised approach using contrastive learning is employed to encode each tool instance into a compact feature vector that captures its semantic content. This is achieved by leveraging intrinsic temporal information from video sequences and local instance tracking to generate positive and negative samples.
- Instance Classification: This module learns to classify tool instances using binary tool presence labels (either frame-wise or sequence-wise), which are much cheaper to obtain than spatial annotations. It uses a teacher-student approach where a teacher network, trained on a small set of human-labeled "prototype instances" (selected via unsupervised clustering), guides the training of a student network on the weak labels.
Datasets
The framework is validated on the following datasets:
- MICCAI 2017 EndoVis Robotic Instrument Segmentation Challenge dataset
- MICCAI 2018 EndoVis Robotic Instrument Segmentation Challenge dataset
Key Insights and Ideas
- Weak Supervision is Powerful: The paper demonstrates that readily available, but spatially unlocalized, binary tool presence labels (frame-wise and sequence-wise) can be effectively used for instance classification, significantly reducing annotation costs.
- Unsupervised Instantiation is Feasible: By combining connected components analysis with a novel instance pasting augmentation, the authors show that it's possible to generate accurate instance masks without ground truth spatial annotations.
- Prototype Instances are Key: A small number of carefully selected "prototype instances" (as few as 8) are sufficient to provide effective supervision for the instance classification stage, highlighting the efficiency of targeted human annotation.
- Sequence-wise Labels are Underrated: The framework effectively utilizes sequence-wise binary tool presence labels, which are even cheaper to obtain than frame-wise labels, showing minimal performance degradation.
Key Results
- Tool Instantiation: The proposed SAF-IS approach outperforms Mask-RCNN in tool instantiation quality across both datasets and various training modalities, especially when using unsupervised binary masks. For example, on EndoVis 2017, SAF-IS achieved a +417.5% improvement in [email protected] when trained with FUN-SIS predicted masks compared to Mask-RCNN.
- Instance Segmentation: SAF-IS, trained with only prototype labels and weak binary tool presence labels, outperforms several state-of-the-art fully-supervised and semi-supervised semantic segmentation methods on the EndoVis datasets. For instance, it achieves competitive results even when compared to methods that utilize temporal information and additional tool-part annotations.
- Annotation Efficiency: The ablation study shows that reducing the number of prototype instances to 8 (approximately 0.26% of training data) results in only a marginal performance drop, underscoring the framework's efficiency.
Limitations and Future Work
- Feature Learning Bottleneck: The paper identifies feature learning as a crucial methodological bottleneck. While currently unsupervised, incorporating weak information during feature learning could further improve performance.
- Temporal Modeling: The authors suggest that temporal modeling, currently used for feature learning, could be directly integrated into the inference stage to further enhance segmentation.
- Leveraging New Segmentation Models: The authors propose that future work could leverage recent advancements like SAM (Segment Anything Model) to obtain initial instance masks, which SAF-IS could then refine and group.
- Directly Using Weak Labels for Feature Learning: The paper suggests that weak labels could be incorporated earlier in the pipeline, specifically during the feature learning stage, to guide positive and negative sample selection.
Summary updated