M2CAI Workflow Challenge: Convolutional Neural Networks with Time Smoothing and Hidden Markov Model for Video Frames Classification

Rémi Cadène, Thomas Robert, Nicolas Thome, Matthieu CordarXiv 2016 · arXiv

Abstract

Automated recognition of surgical phases is a critical component for the development of intelligent surgical assistants and performance evaluation systems. This paper describes a methodology for the M2CAI Workflow challenge, which involves identifying eight distinct phases of a laparoscopic cholecystectomy in real-time. Our approach combines a deep Residual Network (ResNet-200) for visual feature extraction with a Hidden Markov Model (HMM) to enforce temporal consistency. The proposed "online" system achieved a Jaccard index of 71.9 in the competition, placing it among the top three entries.

Introduction

In modern surgical environments, understanding the procedural workflow is essential for optimizing operating room efficiency and providing context-aware assistance. The M2CAI Workflow challenge tasks participants with classifying every frame of endoscopic videos into one of eight phases, such as Trocar Placement or Gallbladder Dissection. A key constraint is the "online" requirement: predictions must be made using only current and past information, simulating a live monitoring scenario.

Methodology

The proposed pipeline consists of two primary stages: visual recognition and temporal smoothing.

Visual Recognition

We evaluated several Convolutional Neural Network (CNN) architectures based on InceptionV3 and ResNet. The models were fine-tuned on the m2cai16-workflow dataset, which contains 27 high-definition videos of cholecystectomies. We found that fine-tuning a pre-trained ResNet-200 model significantly outperformed training from scratch, as demonstrated in the classification model performance comparison.

Comparison of classification models showing the performance gain from fine-tuning and the selection of ResNet-200.

Temporal Smoothing

Raw CNN predictions often exhibit high-frequency "jitter" due to visual similarities between phases. To address this, we implemented a two-step smoothing process. First, log-probabilities were averaged over a sliding window of 15 frames. Second, an HMM was employed to model the logical progression of surgical phases. For online operation, the Viterbi algorithm was applied to the sequence up to the current frame to determine the most likely current state.

Results

The integration of temporal modeling provided a substantial boost in performance over raw CNN outputs. On our internal validation set, the HMM Online approach reached an accuracy of 88.90% and a Jaccard index of 81.60. While an "offline" HMM (which considers the entire video sequence) performs even better, the online version remains robust for real-time applications.

Quantitative comparison of temporal methods, demonstrating the superiority of HMM-based smoothing over simple averaging.

Conclusion

Our results highlight two critical factors for surgical workflow recognition: the superiority of fine-tuning pre-trained models over training from scratch, and the necessity of incorporating temporal "grammar" via HMMs. The HMM effectively prevents impossible state transitions, acting as a mathematical constraint on the surgical sequence. Future work will explore ensemble methods and detailed error analysis to further bridge the gap between online and offline performance.

Summary updated

Abstract

Our approach is among the three best to tackle the M2CAI Workflow challenge. The latter consists in recognizing the operation phase for each frames of endoscopic videos. In this technical report, we compare several classification models and temporal smoothing methods. Our submitted solution is a fine tuned Residual Network-200 on 80% of the training set with temporal smoothing using simple temporal averaging of the predictions and a Hidden Markov Model modeling the sequence.
BibTeX
@misc{cadene2016m,
  author = {Rémi Cadène and Thomas Robert and Nicolas Thome and Matthieu Cord},
  title = {{M2CAI Workflow Challenge: Convolutional Neural Networks with Time Smoothing and Hidden Markov Model for Video Frames Classification}},
  year = {2016},
  eprint = {1610.05541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1610.05541},
}
Figure