Neural Finite-State Machines for Surgical Phase Recognition

Hao Ding, Zhongpai Gao, Benjamin Planche, Tianyu Luan, Abhishek Sharma, Meng Zheng, Ange Lou, Terrence Chen, Mathias Unberath, Ziyan WuarXiv 2024 · DOI · arXiv

Introduction

Surgical Phase Recognition (SPR) is a fundamental task in computer-assisted surgery, involving the automated identification of procedural steps from video streams. Accurate SPR is essential for workflow optimization, surgeon training, and the development of real-time intraoperative guidance systems. However, surgical videos are characterized by long durations and high visual complexity. While state-of-the-art deep learning models, particularly Transformers, excel at frame-level feature extraction, they often struggle with the "big picture" temporal logic of a procedure. This frequently results in "fragmented" or "flickering" predictions, where the model briefly misidentifies a phase due to transient occlusions or atypical movements, violating the inherent structured progression of surgery.

Methodology: The Neural Finite-State Machine (NFSM)

To address these temporal inconsistencies, we introduce the Neural Finite-State Machine (NFSM). This plug-and-play module integrates the logical constraints of classical finite-state machines with the representational power of modern neural architectures.

Detailed architecture of the NFSM module showing the transition from visual embeddings to dynamic transition tables and final predictions.

The NFSM architecture consists of several key components:

  • Learnable Global State Embeddings: The module maintains unique d-dimensional embeddings for each surgical phase, serving as a canonical representation of each state.
  • Dynamic Transition Tables: Unlike static FSMs, NFSM utilizes an attention-based mechanism to generate transition tables ($T_t$) dynamically. These tables estimate the probability of transitioning between states based on the current visual context.
  • Future Phase Forecasting: To enhance temporal awareness, the model employs "Future Forecasting Blocks." By padding input sequences with repetitions of current frames (pseudo-future embeddings), the model learns to anticipate upcoming transitions.
  • Transition-Aware Inference: The final prediction ($P_t$) is not derived solely from the current frame. Instead, the module aggregates transition probabilities over a historical window, effectively filtering out logically improbable phase jumps and smoothing the output.

Experimental Evaluation

The NFSM module was evaluated across three diverse surgical benchmarks:

  1. BernBypass70: A challenging dataset of 70 gastric bypass procedures, characterized by long durations (avg. 72 mins) and complex workflows.
  2. Cholec80: A standard benchmark consisting of 80 cholecystectomy (gallbladder removal) videos.
  3. AutoLaparo: A dataset of 21 laparoscopic hysterectomy videos, used to assess the model's cross-procedural generalizability.

Results and Discussion

The integration of NFSM consistently improved performance across all benchmarks. On the BernBypass70 dataset, the module increased video-level accuracy by 0.9 points and significantly boosted phase-level precision and mean Average Precision (mAP) by 3.8 and 4.1 points, respectively.

Qualitative comparison of phase recognition timelines, highlighting the reduction in prediction fragmentation when using NFSM.

Qualitative analysis (see Figure above) demonstrates that NFSM drastically reduces "flickering" errors. While baseline models often produce fragmented predictions, the addition of NFSM results in continuous, coherent phase blocks that closely align with the ground-truth annotations. Furthermore, ablation studies revealed that NFSM provides performance gains even when the underlying vision backbone is "frozen," suggesting its utility as a robust adapter for large-scale surgical foundation models.

Conclusion

The Neural Finite-State Machine represents a significant step toward more reliable surgical AI by enforcing procedural logic within a deep learning framework. By bridging the gap between flexible neural networks and structured domain knowledge, NFSM provides a scalable solution for reducing prediction noise in complex clinical environments. Future research will focus on adapting the dynamic transition mechanism to better handle non-linear workflows and unexpected surgical complications.

Summary updated

Abstract

Surgical phase recognition (SPR) is crucial for applications in workflow optimization, performance evaluation, and real-time intervention guidance. However, current deep learning models often struggle with fragmented predictions, failing to capture the sequential nature of surgical workflows. We propose the Neural Finite-State Machine (NFSM), a novel approach that enforces temporal coherence by integrating classical state-transition priors with modern neural networks. NFSM leverages learnable global state embeddings as unique phase identifiers and dynamic transition tables to model phase-to-phase progressions. Additionally, a future phase forecasting mechanism employs repeated frame padding to anticipate upcoming transitions. Implemented as a plug-and-play module, NFSM can be integrated into existing SPR pipelines without changing their core architectures. We demonstrate state-of-the-art performance across multiple benchmarks, including a significant improvement on the BernBypass70 dataset - raising video-level accuracy by 0.9 points and phase-level precision, recall, F1-score, and mAP by 3.8, 3.1, 3.3, and 4.1, respectively. Ablation studies confirm each component's effectiveness and the module's adaptability to various architectures. By unifying finite-state principles with deep learning, NFSM offers a robust path toward consistent, long-term surgical video analysis.
BibTeX
@misc{ding2024neural,
  author = {Hao Ding and Zhongpai Gao and Benjamin Planche and Tianyu Luan and Abhishek Sharma and Meng Zheng and Ange Lou and Terrence Chen and Mathias Unberath and Ziyan Wu},
  title = {{Neural Finite-State Machines for Surgical Phase Recognition}},
  year = {2024},
  eprint = {2411.18018},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.18018},
}
Figure