Temporal Coherence-based Self-supervised Learning for Laparoscopic Workflow Analysis

Isabel Funke, Alexander Jenke, Sören Torge Mees, Jürgen Weitz, Stefanie Speidel, Sebastian Bodenstedt

TL;DR

This paper introduces a self-supervised learning approach for surgical workflow analysis that leverages the "temporal coherence" of video—the principle that consecutive frames should look similar and change steadily. By pretraining neural networks on unlabeled laparoscopic videos using these inherent temporal cues, the researchers significantly improved surgical phase recognition, achieving a 10-point $F_1$ score boost when labeled data was scarce.

What Problem Does This Paper Address?

Computer-assisted surgery (CAS) systems need to understand the "context" of an operation—knowing exactly which phase of a procedure is happening—to provide the right assistance at the right time. While deep learning models like CNN-LSTMs are great at this, they are "data-hungry."

In the medical field, annotating thousands of hours of surgical video is incredibly expensive and requires expert clinicians' time. This paper addresses the "small data" bottleneck by finding a way to train models on the vast amounts of unlabeled surgical video already available, making the subsequent supervised learning much more efficient.

Methods

The authors propose using Self-Supervised Learning (SSL) based on Slow Feature Analysis (SFA). The core idea is that in a video, frames that are close in time should have similar mathematical representations (embeddings), while frames far apart should be different.

They used a ResNet-50 architecture and tested three different self-supervised "proxy tasks" for pretraining:

  1. Contrastive Loss: Forces the network to map temporally close frames to similar points in feature space and distant frames to points at least a certain distance apart.
  2. Ranking Loss: A triplet-based approach that ensures a "close" frame is always more similar to the query frame than a "distant" frame is.
  3. 1st & 2nd Order Contrastive Loss: This is the most novel addition. It enforces both "slowness" (frames change slowly) and "steadiness" (the rate of change between frames is consistent, meaning abrupt, erratic motions are penalized in the learned features).

After pretraining the CNN on unlabeled data, they added a Long Short-Term Memory (LSTM) unit to handle the temporal sequence and fine-tuned the entire "PhaseNet" on a smaller set of labeled data.

Datasets

The study utilized Cholec80, a standard public dataset containing 80 videos of laparoscopic cholecystectomy (gallbladder removal) procedures.

  • Pretraining: Used 60 videos, ignoring the labels.
  • Fine-tuning/Evaluation: Used subsets of 20, 40, and 60 labeled videos to test how well the pretraining helps when data is limited.
  • Testing: A held-out set of 20 videos.

Key Insights and Ideas

  • Temporal Coherence as a "Free" Label: You don't need a human to tell a computer that frame 100 is probably more like frame 101 than frame 5000. This inherent structure of video provides a massive, free supervisory signal.
  • The Power of "Steady" Features: The 2nd-order coherence (steadiness) proved particularly effective. It encourages the model to ignore "noise" like camera shakes or minor instrument jitters and focus on the underlying, consistent progression of the surgery.
  • Efficiency Gains: The most striking insight is that a model pretrained with SSL using only 40 labeled videos outperformed a standard model trained on 60 labeled videos. This suggests SSL can effectively "substitute" for a significant amount of manual annotation.
  • Semantic Emergence: Even though the network was never told what a "tool" or "organ" was during pretraining, visual inspections of the feature space showed that the model naturally grouped images with similar anatomical structures and instrument configurations together.

Key Results

  • Performance Boost: When only 20 labeled videos were available, the $F_1$ score jumped from 67.8 (no pretraining) to 78.6 (with 1st & 2nd order contrastive pretraining).
  • Top Performance: The best-performing model reached a maximum $F_1$ score of 84.6 on the surgical phase segmentation task.
  • Consistency: Pretraining not only improved accuracy but also halved the standard deviation of the results, meaning the models became much more reliable across different surgical cases.

Limitations and Future Work

  • Short Phases: The method struggled with very short surgical phases (like Phase 7, which lasts only 1-3 minutes). Because these phases are so brief, the "temporally distant" pairs used in training might actually belong to the same phase, or "close" pairs might cross phase boundaries, confusing the model.
  • Visual Ambiguity: Some phases are visually similar but semantically different. The current temporal coherence approach relies purely on visual similarity, which might not be enough to distinguish between complex, look-alike surgical steps.
  • Future Directions: The authors suggest exploring whether these learned embeddings can be used for completely unsupervised detection of surgical steps or as a regularization tool during standard supervised training.

Summary updated

Abstract

In order to provide the right type of assistance at the right time, computer-assisted surgery systems need context awareness. To achieve this, methods for surgical workflow analysis are crucial. Currently, convolutional neural networks provide the best performance for video-based workflow analysis tasks. For training such networks, large amounts of annotated data are necessary. However, collecting a sufficient amount of data is often costly, time-consuming, and not always feasible. In this paper, we address this problem by presenting and comparing different approaches for self-supervised pretraining of neural networks on unlabeled laparoscopic videos using temporal coherence. We evaluate our pretrained networks on Cholec80, a publicly available dataset for surgical phase segmentation, on which a maximum $$F_1$$ score of 84.6 was reached. Furthermore, we were able to achieve an increase of the $$F_1$$ score of up to 10 points when compared to a non-pretrained neural network.
BibTeX
@inproceedings{funke2018temporal,
  author = {Isabel Funke and Alexander Jenke and Sören Torge Mees and Jürgen Weitz and Stefanie Speidel and Sebastian Bodenstedt},
  title = {{Temporal Coherence-based Self-supervised Learning for Laparoscopic Workflow Analysis}},
  booktitle = {OR 2.0/CARE/CLIP/ISIC @ MICCAI},
  year = {2018},
  url = {https://openreview.net/forum?id=sTdg0Gai5GuC},
}

Loading related content...

Figure