Surgical Phase Recognition in Laparoscopic Cholecystectomy

Yunfan Li, Vinayak Shenoy, Prateek Prasanna, I. V. Ramakrishnan, Haibin Ling, Himanshu Gupta

TL;DR

This paper introduces a two-stage inference pipeline for surgical phase recognition that dynamically switches between a general baseline model and specialized "transition" models. By applying temperature scaling to calibrate the Transformer's confidence scores, the authors effectively mitigate over-confidence issues and improve phase recognition accuracy in laparoscopic surgery videos.

What Problem Does This Paper Address?

In the world of robot-assisted surgery, "surgical workflow analysis"—knowing exactly which step of a procedure is happening at any given moment—is vital for training, intraoperative assistance, and safety. However, automatically recognizing these phases from video is difficult.

While modern Transformer models are excellent at capturing long-term temporal relationships, they face two major hurdles:

  1. Boundary Noise: Predictions often "flicker" or jump back and forth during the transition between two surgical steps.
  2. Miscalibration: Deep learning models, especially Transformers, tend to be "over-confident." They might predict the wrong phase with a very high probability score, making it hard for a system to know when to trust the model or when to seek a second opinion.

Methods

The authors propose a system that doesn't just rely on one "master" model. Instead, they use a hierarchical approach:

  • The Baseline: They adopt Trans-SVNet, a state-of-the-art Transformer-based network (see Figure 1), as their primary 7-class classifier.
  • Transition Models: They train six separate 2-class classifiers, each specialized in distinguishing between two neighboring phases (e.g., Phase 1 vs. Phase 2).
  • Confidence-Based Inference: During a surgery, the system monitors the baseline model's confidence. If the confidence score drops below a certain threshold ($t_{conf}$), the system assumes the surgery is at a transition point and switches to the relevant 2-class transition model to make the final call.
  • Temperature Scaling: To fix the over-confidence issue, they apply a calibration technique called temperature scaling. This adjusts the model's output logits using a single scalar parameter ($T$) before the final softmax layer, ensuring the "confidence" actually reflects the likelihood of being correct.
Fig. 1. Transformer Model

Datasets

The study uses the Cholec80 dataset, the industry standard for laparoscopic cholecystectomy (gallbladder removal) research.

  • Content: 80 videos recorded at 25 fps, defining 7 distinct surgical stages.
  • Processing: Videos were subsampled to 1 fps and resized to 250x250 pixels.
  • Split: 40 videos for training, 8 for validation, and 32 for testing.

Key Insights and Ideas

  • Binary is Better at the Border: The authors found that 2-class classifiers are significantly more accurate at distinguishing between two sequential steps than a full 7-class model. As shown in Table 1, most transition models achieved mid-to-high 90% accuracy, whereas the baseline was around 87%.
  • The Danger of "Cascading Effects": An early attempt to switch models based on a simple majority-vote buffer failed. If the model made a streak of errors, it would get "stuck" in the wrong transition model for the rest of the video, leading to catastrophic failure (see Figure 2).
  • Calibration is the Secret Sauce: Without temperature scaling, the confidence-based strategy performed poorly (65.64% accuracy) because the model was "loudly wrong." Calibration brought the accuracy up to 88.02%, proving that reliable confidence scores are essential for multi-model pipelines.
An illustration of the cascading effect, where a wrong model choice leads to a long streak of incorrect predictions compared to the ground truth.

Key Results

The proposed calibrated, confidence-based inference strategy outperformed the baseline and other naive switching methods.

Model Accuracy (%)
Trans-SVNet (Baseline) 87.44
Transition-based (Buffer) 86.92
Confidence-based (Uncalibrated) 65.64
Confidence-based (Calibrated) 88.02

The calibration itself was highly effective; temperature scaling reduced the Expected Calibration Error (ECE) from 0.215 down to 0.031, meaning the model's reported probabilities became much more representative of real-world accuracy.

Limitations and Future Work

While the method improves accuracy, the authors note that local transitional constraints can still lead to cascading effects if the initial phase detection is wrong. The system relies heavily on the surgery following a strictly linear sequence of steps, which may not always happen in complex or emergency cases.

Future work could involve applying this confidence-based switching to other types of action segmentation or exploring more robust ways to recover from the "cascading effect" when a transition model is triggered prematurely.

Summary updated

Abstract

Automatic recognition of surgical phases in surgical videos is a fundamental task in surgical workflow analysis. In this report, we propose a Transformer-based method that utilizes calibrated confidence scores for a 2-stage inference pipeline, which dynamically switches between a baseline model and a separately trained transition model depending on the calibrated confidence level. Our method outperforms the baseline model on the Cholec80 dataset, and can be applied to a variety of action segmentation methods.
BibTeX
@inproceedings{li2022surgical,
  author = {Yunfan Li and Vinayak Shenoy and Prateek Prasanna and I. V. Ramakrishnan and Haibin Ling and Himanshu Gupta},
  title = {{Surgical Phase Recognition in Laparoscopic Cholecystectomy}},
  booktitle = {CENTERIS/ProjMAN/HCist},
  year = {2022},
}
Figure