Temporal coherence-based self-supervised learning for laparoscopic workflow analysis

Isabel Funke, Alexander Jenke, Sören Torge Mees, Jürgen Weitz, Stefanie Speidel, Sebastian Bodenstedt

Abstract In order to provide the right type of assistance at the right time, computer-assisted surgery systems need context awareness. To achieve this, methods for surgical workflow analysis are crucial. Currently, convolutional neural networks provide the best performance for video-based workflow analysis tasks. For training such networks, large amounts of annotated data are necessary. However, collecting a sufficient amount of data is often costly, time-consuming, and not always feasible. In this paper, we address this problem by presenting and comparing different approaches for self-supervised pretraining of neural networks on unlabeled laparoscopic videos using temporal coherence. We evaluate our pretrained networks on Cholec80, a publicly available dataset for surgical phase segmentation, on which a maximum F1 score of 84.6 was reached. Furthermore, we were able to achieve an increase of the F1 score of up to 10 points when compared to a non-pretrained neural network.

Introduction

Computer-assisted surgery (CAS) systems aim to enhance surgical precision and safety by providing real-time context-aware assistance. A fundamental requirement for such systems is the ability to recognize the current phase of a surgical procedure. While deep learning models, particularly CNN-LSTM architectures, have shown state-of-the-art performance in surgical phase recognition, their efficacy is heavily dependent on large, expertly annotated datasets. In the medical domain, obtaining such labels is prohibitively expensive and time-consuming.

This research addresses the "small data" challenge by leveraging self-supervised learning (SSL). By utilizing the inherent temporal structure of unlabeled surgical videos—a concept known as temporal coherence—we can pretrain models to learn meaningful visual representations before any manual labels are introduced.

Methods: Leveraging Temporal Coherence

The core hypothesis of this work is that consecutive video frames share high semantic similarity, and visual features change slowly and steadily over time. This principle, derived from Slow Feature Analysis (SFA), allows for the creation of "proxy tasks" that do not require human labels.

Visualizing temporal coherence in laparoscopic video frames across different time intervals.

We utilized a ResNet-50 architecture to evaluate three distinct SSL proxy tasks:

  1. Contrastive Loss: This task forces the network to map temporally adjacent frames to similar points in a feature space while pushing distant frames apart.
  2. Ranking Loss: A triplet-based approach ensuring that a frame close in time to a query frame is represented more similarly than a distant frame.
  3. 1st & 2nd Order Contrastive Loss: This novel approach enforces both "slowness" (slow feature changes) and "steadiness" (consistent rate of change). By penalizing abrupt or erratic feature transitions, the model learns to prioritize consistent surgical progression over transient noise like camera jitter.

Following pretraining on unlabeled data, a Long Short-Term Memory (LSTM) unit was integrated to capture long-range temporal dependencies, and the full model (PhaseNet) was fine-tuned on varying amounts of labeled data.

Experimental Setup

The models were evaluated using the Cholec80 dataset, which contains 80 videos of laparoscopic cholecystectomy procedures. Pretraining was conducted on 60 videos without using their labels. To test the efficiency of SSL under data-constrained scenarios, fine-tuning was performed on subsets of 20, 40, and 60 labeled videos. A separate set of 20 videos was reserved for final testing.

Key Results and Discussion

The experimental results demonstrate that self-supervised pretraining significantly enhances performance, particularly when labeled data is scarce.

Performance comparison of different pretraining strategies across varying amounts of labeled training data (#OPs).

As shown in the table above, when only 20 labeled videos were available, the $F_1$ score improved from 67.8 (no pretraining) to 78.6 using 1st & 2nd order contrastive pretraining. Notably, a model pretrained with SSL using only 40 labeled videos outperformed a standard model trained on 60 labeled videos, suggesting that SSL can effectively substitute for a significant portion of manual annotation effort.

Beyond raw accuracy, pretraining improved the reliability of the models, halving the standard deviation of results across different cases. Visual inspection of the learned feature space further revealed that the model naturally grouped images with similar anatomical structures and instrument configurations, even without explicit supervision.

Conclusion and Limitations

Self-supervised learning based on temporal coherence offers a robust solution to the data bottleneck in surgical workflow analysis. By enforcing "steadiness" in feature learning, we achieved a peak $F_1$ score of 84.6.

However, limitations remain. The method currently struggles with very short surgical phases (e.g., Phase 7, which may last only 1-3 minutes), as the temporal sampling logic may inadvertently group frames from different phases or separate frames that should be linked. Future work will explore using these learned embeddings for fully unsupervised surgical step detection and as a regularization mechanism during standard supervised training.

Summary updated

Abstract

In order to provide the right type of assistance at the right time, computer-assisted surgery systems need context awareness. To achieve this, methods for surgical workflow analysis are crucial. Currently, convolutional neural networks provide the best performance for video-based workflow analysis tasks. For training such networks, large amounts of annotated data are necessary. However, collecting a sufficient amount of data is often costly, time-consuming, and not always feasible. In this paper, we address this problem by presenting and comparing different approaches for self-supervised pretraining of neural networks on unlabeled laparoscopic videos using temporal coherence. We evaluate our pretrained networks on Cholec80, a publicly available dataset for surgical phase segmentation, on which a maximum F1 score of 84.6 was reached. Furthermore, we were able to achieve an increase of the F1 score of up to 10 points when compared to a non-pretrained neural network.
BibTeX
@inproceedings{funke2018temporal,
  author = {Isabel Funke and Alexander Jenke and Sören Torge Mees and Jürgen Weitz and Stefanie Speidel and Sebastian Bodenstedt},
  title = {{Temporal coherence-based self-supervised learning for laparoscopic workflow analysis}},
  booktitle = {OR 2.0/CARE/CLIP/ISIC @ MICCAI},
  year = {2018},
  url = {https://openreview.net/forum?id=sTdg0Gai5GuC},
}
Figure