Introduction
Surgical Phase Recognition (SPR) is a fundamental task in computer-assisted surgery, involving the automated identification of procedural steps from video streams. Accurate SPR is essential for workflow optimization, surgeon training, and the development of real-time intraoperative guidance systems. However, surgical videos are characterized by long durations and high visual complexity. While state-of-the-art deep learning models, particularly Transformers, excel at frame-level feature extraction, they often struggle with the "big picture" temporal logic of a procedure. This frequently results in "fragmented" or "flickering" predictions, where the model briefly misidentifies a phase due to transient occlusions or atypical movements, violating the inherent structured progression of surgery.
Methodology: The Neural Finite-State Machine (NFSM)
To address these temporal inconsistencies, we introduce the Neural Finite-State Machine (NFSM). This plug-and-play module integrates the logical constraints of classical finite-state machines with the representational power of modern neural architectures.
The NFSM architecture consists of several key components:
- Learnable Global State Embeddings: The module maintains unique d-dimensional embeddings for each surgical phase, serving as a canonical representation of each state.
- Dynamic Transition Tables: Unlike static FSMs, NFSM utilizes an attention-based mechanism to generate transition tables ($T_t$) dynamically. These tables estimate the probability of transitioning between states based on the current visual context.
- Future Phase Forecasting: To enhance temporal awareness, the model employs "Future Forecasting Blocks." By padding input sequences with repetitions of current frames (pseudo-future embeddings), the model learns to anticipate upcoming transitions.
- Transition-Aware Inference: The final prediction ($P_t$) is not derived solely from the current frame. Instead, the module aggregates transition probabilities over a historical window, effectively filtering out logically improbable phase jumps and smoothing the output.
Experimental Evaluation
The NFSM module was evaluated across three diverse surgical benchmarks:
- BernBypass70: A challenging dataset of 70 gastric bypass procedures, characterized by long durations (avg. 72 mins) and complex workflows.
- Cholec80: A standard benchmark consisting of 80 cholecystectomy (gallbladder removal) videos.
- AutoLaparo: A dataset of 21 laparoscopic hysterectomy videos, used to assess the model's cross-procedural generalizability.
Results and Discussion
The integration of NFSM consistently improved performance across all benchmarks. On the BernBypass70 dataset, the module increased video-level accuracy by 0.9 points and significantly boosted phase-level precision and mean Average Precision (mAP) by 3.8 and 4.1 points, respectively.
Qualitative analysis (see Figure above) demonstrates that NFSM drastically reduces "flickering" errors. While baseline models often produce fragmented predictions, the addition of NFSM results in continuous, coherent phase blocks that closely align with the ground-truth annotations. Furthermore, ablation studies revealed that NFSM provides performance gains even when the underlying vision backbone is "frozen," suggesting its utility as a robust adapter for large-scale surgical foundation models.
Conclusion
The Neural Finite-State Machine represents a significant step toward more reliable surgical AI by enforcing procedural logic within a deep learning framework. By bridging the gap between flexible neural networks and structured domain knowledge, NFSM provides a scalable solution for reducing prediction noise in complex clinical environments. Future research will focus on adapting the dynamic transition mechanism to better handle non-linear workflows and unexpected surgical complications.
Summary updated