Abstract
Automated recognition of surgical phases is a critical component for the development of intelligent surgical assistants and performance evaluation systems. This paper describes a methodology for the M2CAI Workflow challenge, which involves identifying eight distinct phases of a laparoscopic cholecystectomy in real-time. Our approach combines a deep Residual Network (ResNet-200) for visual feature extraction with a Hidden Markov Model (HMM) to enforce temporal consistency. The proposed "online" system achieved a Jaccard index of 71.9 in the competition, placing it among the top three entries.
Introduction
In modern surgical environments, understanding the procedural workflow is essential for optimizing operating room efficiency and providing context-aware assistance. The M2CAI Workflow challenge tasks participants with classifying every frame of endoscopic videos into one of eight phases, such as Trocar Placement or Gallbladder Dissection. A key constraint is the "online" requirement: predictions must be made using only current and past information, simulating a live monitoring scenario.
Methodology
The proposed pipeline consists of two primary stages: visual recognition and temporal smoothing.
Visual Recognition
We evaluated several Convolutional Neural Network (CNN) architectures based on InceptionV3 and ResNet. The models were fine-tuned on the m2cai16-workflow dataset, which contains 27 high-definition videos of cholecystectomies. We found that fine-tuning a pre-trained ResNet-200 model significantly outperformed training from scratch, as demonstrated in the classification model performance comparison.
Temporal Smoothing
Raw CNN predictions often exhibit high-frequency "jitter" due to visual similarities between phases. To address this, we implemented a two-step smoothing process. First, log-probabilities were averaged over a sliding window of 15 frames. Second, an HMM was employed to model the logical progression of surgical phases. For online operation, the Viterbi algorithm was applied to the sequence up to the current frame to determine the most likely current state.
Results
The integration of temporal modeling provided a substantial boost in performance over raw CNN outputs. On our internal validation set, the HMM Online approach reached an accuracy of 88.90% and a Jaccard index of 81.60. While an "offline" HMM (which considers the entire video sequence) performs even better, the online version remains robust for real-time applications.
Conclusion
Our results highlight two critical factors for surgical workflow recognition: the superiority of fine-tuning pre-trained models over training from scratch, and the necessity of incorporating temporal "grammar" via HMMs. The HMM effectively prevents impossible state transitions, acting as a mathematical constraint on the surgical sequence. Future work will explore ensemble methods and detailed error analysis to further bridge the gap between online and offline performance.
Summary updated