Surgical Phase Recognition in Laparoscopic Cholecystectomy

Yunfan Li, Himanshu Gupta, Prateek Prasanna, I. V. Ramakrishnan, Haibin Ling

TL;DR

This paper tackles the challenge of accurately recognizing surgical phases in laparoscopic cholecystectomy videos. They propose a novel Transformer-based inference strategy that dynamically switches between a general phase predictor and specialized 2-class predictors based on the confidence of the initial prediction, significantly improving accuracy by addressing over-confident model outputs.

What Problem Does This Paper Address?

Surgical workflow analysis, particularly automatic segmentation of surgical steps, is crucial for improving patient safety, surgical training, and intraoperative assistance. While Transformer models have shown promise in this area, they often suffer from over-confident predictions, meaning they can be highly certain about incorrect classifications. This miscalibration hinders their reliability in safety-critical applications where knowing when a model is likely wrong is as important as making correct predictions. Existing methods struggle to effectively leverage the sequential nature of surgical procedures and the distinctiveness of transitions between specific surgical phases.

Methods

The core of this work is a new inference strategy built upon a Transformer-based baseline model (Trans-SVNet). Instead of relying solely on the baseline for predicting one of seven surgical phases, the authors introduce a two-stage approach:

  1. Transition Models: They train specialized 2-class Transformer models for each adjacent pair of surgical phases (e.g., Phase 1 to Phase 2, Phase 2 to Phase 3). The intuition is that distinguishing between two specific phases is easier than classifying among seven.
  2. Confidence-Based Inference: This is the key innovation. The strategy uses the confidence score from the baseline model's prediction.
  3. If the baseline model is highly confident (confidence above a threshold tconf), its prediction is used.
  4. If the baseline model is less confident, the system switches to a relevant 2-class transition model. The specific transition model used depends on the previous predicted phase. For instance, if the previous phase was predicted as 3, and the current prediction is uncertain, the system uses the "Trans3-4" model to predict the next phase.
  5. Confidence Calibration: To combat the inherent over-confidence of deep learning models, they employ temperature scaling. This technique adjusts the model's output logits before the softmax function, effectively "softening" the probability distribution and providing more reliable confidence scores. This calibrated confidence is then used in the confidence-based inference strategy.

Datasets

The study utilizes the Cholec80 dataset, which contains 80 videos of laparoscopic cholecystectomy procedures. The videos are preprocessed by subsampling to 1 fps and resizing frames to 250x250 pixels. The dataset is split into 40 videos for training, 8 for validation, and 32 for testing.

Key Insights and Ideas

  • 2-class classifiers are better for transitions: The paper empirically demonstrates that specialized 2-class classifiers can achieve higher accuracy in distinguishing between adjacent surgical phases compared to a single 7-class classifier.
  • Over-confidence is a major hurdle: The baseline Transformer model, despite its high accuracy, often produces over-confident predictions, making it unreliable when it's wrong.
  • Dynamic switching based on confidence is effective: By intelligently switching to specialized 2-class models only when the main model is uncertain, the system can leverage the strengths of both approaches.
  • Calibration is crucial for confidence-based methods: The confidence-based inference strategy's success hinges on accurate confidence scores. Temperature scaling proved essential for making these scores more reliable and improving overall performance.
  • Cascading effects can be detrimental: The paper highlights that a simple transition-based inference (without confidence calibration) can lead to a "cascading effect," where an initial incorrect prediction can lead to a chain of subsequent incorrect predictions, worsening overall accuracy.

Key Results

  • The baseline Trans-SVNet model achieved an accuracy of 87.44%.
  • A simple transition-based inference strategy (without confidence calibration) resulted in a slight decrease in accuracy to 86.92%, illustrating the cascading effect.
  • The proposed confidence-based inference strategy with temperature scaling achieved the highest accuracy of 88.02%, outperforming the baseline model.
  • Confidence calibration significantly improved metrics like Negative Log-Likelihood (NLL) from 0.576 to 0.402 and Expected Calibration Error (ECE) from 0.215 to 0.031, demonstrating better calibrated models.

Limitations and Future Work

  • Cascading Effect: While the confidence-based strategy mitigates the cascading effect seen in simpler transition-based methods, the paper acknowledges that it's a potential issue that needs careful management.
  • Generalizability: The proposed method is evaluated on a specific surgical procedure (laparoscopic cholecystectomy). Future work could explore its applicability to other surgical domains.
  • Transition Model Training: The training of 2-class transition models is done independently. Investigating methods to jointly train or fine-tune these models could potentially yield further improvements.
  • Threshold Optimization: The confidence threshold (tconf) is a hyperparameter. Exploring adaptive or learned thresholds could enhance robustness.

Summary updated

Abstract

Automatic recognition of surgical phases in surgical videos is a fundamental task in surgical workflow analysis. Previous works on automated recognition of surgical phases utilized popular methods in computer vision but failed to consider the sequential nature of surgical procedures. In this paper, we propose a method that utilizes calibrated confidence scores to dynamically switch between two Transformer-based models, viz., baseline and a separately trained binary classifier model, depending on the calibrated confidence level. Our method outperforms the baseline model on the publicly available Cholec80 dataset and can be readily applied to a variety of phase recognition methods and applications.
BibTeX
@inproceedings{li2022surgical,
  author = {Yunfan Li and Himanshu Gupta and Prateek Prasanna and I. V. Ramakrishnan and Haibin Ling},
  title = {{Surgical Phase Recognition in Laparoscopic Cholecystectomy}},
  booktitle = {CENTERIS/ProjMAN/HCist},
  year = {2022},
}
Figure