Self-supervised learning via cluster distance prediction for operating room context awareness

Idris Hamoud, Alexandros Karargyris, Aidean Sharghi, Omid Mohareri, Nicolas PadoyInt. J. Comput. Assist. Radiol. Surg 2022 · DOI · OpenReview

Abstract

Semantic segmentation and activity classification are key components to creating intelligent surgical systems able to understand and assist clinical workflow. In the Operating Room, semantic segmentation is at the core of creating robots aware of clinical surroundings, whereas activity classification aims at understanding OR workflow at a higher level. State-of-the-art semantic segmentation and activity recognition approaches are fully supervised, which is not scalable. Self-supervision can decrease the amount of annotated data needed. We propose a new 3D self-supervised task for OR scene understanding utilizing OR scene images captured with ToF cameras. Contrary to other self-supervised approaches, where handcrafted pretext tasks are focused on 2D image features, our proposed task consists of predicting the relative 3D distance of image patches by exploiting the depth maps. Learning 3D spatial context generates discriminative features for our downstream tasks. Our approach is evaluated on two tasks and datasets containing multi-view data captured from clinical scenarios. We demonstrate a noteworthy improvement of performance on both tasks, specifically on low-regime data where utility of self-supervised learning is the highest.

Introduction

Developing intelligent surgical assistants requires AI models capable of understanding the complex context of an Operating Room (OR). This involves two primary tasks: semantic segmentation (identifying objects like surgical robots and tables) and activity classification (recognizing the current phase of the surgical workflow). Traditionally, these models rely on fully supervised learning, which necessitates vast amounts of expert-labeled data—a significant bottleneck in medical domains. Furthermore, the use of standard RGB video raises substantial privacy concerns for both patients and medical staff.

While self-supervised learning (SSL) has proven effective in general computer vision by using "pretext tasks" to learn features without manual labels, most existing methods focus on 2D image properties. This research addresses the need for 3D spatial awareness in the OR by proposing a novel pretext task specifically designed for depth-based scene understanding.

Methods

The core of the proposed approach is a 3D-based pretext task called Cluster Distance Prediction. By utilizing depth maps from Time-of-Flight (ToF) cameras instead of RGB images, the method inherently preserves privacy while capturing essential spatial information.

Pretext Task Pipeline

The process involves three main stages:

  1. Superpixel Clustering: Pixels are grouped into "superpixels" using the SLIC algorithm to identify homogeneous regions likely belonging to the same object.
  2. 3D Mapping and Filtering: 2D superpixels are mapped into 3D space using camera intrinsics. Heuristics such as convexity and depth disparity are applied to filter out unreliable clusters.
  3. Distance Prediction: The model is trained to predict the relative 3D Euclidean distance between the centroids of pairs of these superpixel regions.
Overview of the 3D pretext task generation process, from superpixel clustering to 3D distance pairing.

Architecture

The model utilizes a ResNet-based encoder-decoder architecture. The Superpixel-based Self-supervised (SPS) module extracts features from the feature map corresponding to the selected superpixels and computes the L2 distance, which is then compared against the ground truth 3D distance derived from the depth map.

The proposed network architecture featuring an encoder-decoder backbone and the Superpixel-based Self-supervised (SPS) module.

By predicting distances, the model learns viewpoint-invariant features, as the physical distance between objects remains constant regardless of camera placement—a vital property for multi-camera OR environments.

Datasets

The study evaluates the method on two distinct datasets:

  • Semantic Segmentation Dataset: Comprises 7,980 densely annotated depth images from four ToF cameras mounted on a da Vinci surgical robot, covering eight object classes (e.g., OR table, Patient Side Cart, Human).
  • Activity Classification Dataset: Includes 98,929 frames from 34 surgical cases, covering ten clinically relevant activities such as "Patient Roll In" and "Da Vinci Surgery."

Results

The proposed SSL method was compared against training from scratch and standard SSL baselines (RotNet and CPC v2). Key findings include:

  • Performance Gains: The method consistently outperformed training from scratch across different ResNet architectures.
  • Low-Data Efficiency: In scenarios with limited labeled data (2% to 10% of the dataset), the method showed improvements of up to 20% for semantic segmentation and 10% for activity classification over baselines.
  • Annotation Reduction: For activity classification, the SSL-pretrained model achieved performance parity with a fully supervised model while using only half the amount of annotated data.

Conclusion

This research demonstrates that predicting 3D distances between clusters in depth maps is a powerful pretext task for OR scene understanding. By focusing on 3D spatial context, the model develops a "common sense" of the physical layout that significantly boosts performance, especially when labels are scarce. Future work will explore incorporating temporal context to further enhance activity recognition and automating the heuristic-based cluster selection process to improve robustness.

Summary updated

Abstract

Semantic segmentation and activity classification are key components to create intelligent surgical systems able to understand and assist clinical workflow. In the operating room, semantic segmentation is at the core of creating robots aware of clinical surroundings, whereas activity classification aims at understanding OR workflow at a higher level. State-of-the-art semantic segmentation and activity recognition approaches are fully supervised, which is not scalable. Self-supervision can decrease the amount of annotated data needed.
BibTeX
@article{hamoud2022self,
  author = {Idris Hamoud and Alexandros Karargyris and Aidean Sharghi and Omid Mohareri and Nicolas Padoy},
  title = {{Self-supervised learning via cluster distance prediction for operating room context awareness}},
  journal = {Int. J. Comput. Assist. Radiol. Surg.},
  year = {2022},
  doi = {10.1007/s11548-022-02629-9},
}
Figure