Surgical procedures are highly structured yet dynamic events. Inside the operating room, a surgeon's actions follow a strict procedural logic, but the visual environment is chaotic—filled with deformable anatomy, shifting illumination, fluid obstructions, and tool occlusions.
Automated analysis of surgical videos aims to translate these raw pixel streams into structured, semantic data. By teaching machine learning models to recognize what is happening in an operating room, researchers are laying the groundwork for real-time intraoperative guidance, objective skill assessment, and automated post-operative documentation.
The Hierarchy of Surgical Workflows
To understand a surgery, a computer cannot treat every frame in isolation. Surgical workflows are inherently hierarchical, spanning multiple granularities of time and intent:
- Phases: High-level blocks of the procedure (e.g., dissection, suturing).
- Steps/Tasks: Specific procedural goals within a phase.
- Gestures/Activities: Atomic physical movements made by the surgeon or instruments.
The core intuition behind modern workflow recognition is that these levels are deeply interdependent. For instance, knowing that a surgeon is currently in a "suturing" phase heavily constrains the probability of what the next gesture will be.
To capture this structure, Yanzhe Liu et al. developed a hierarchical model in Multilevel effective surgical workflow recognition in robotic left lateral sectionectomy with deep learning: experimental research. A key insight from their work is the concept of filtering out "under-effective" or non-essential frames. By training models to focus primarily on frames containing active surgical interactions, the signal-to-noise ratio improves dramatically, allowing the network to map long-term temporal dependencies more accurately.
Similarly, Chengbo Zang et al. demonstrated in Surgical Phase Recognition in Inguinal Hernia Repair—AI-Based Confirmatory Baseline and Exploration of Competitive Models that advanced architectures like Video Swin Transformers and Perceiver IO excel at this task. These models succeed because they use attention mechanisms to weigh spatial features (like the presence of a specific mesh) alongside temporal features (the sequence of actions leading up to mesh placement), mirroring how a human expert identifies procedural transitions.
[Raw Video Stream]
│
▼
[Spatial Feature Extraction] ──► Identifies tools, anatomy, and fluids
│
▼
[Temporal Modeling] ──► Captures sequential logic over time
│
├─► Phase Recognition (e.g., Dissection vs. Anastomosis)
├─► Step Identification (e.g., Suture placement)
└─► Gesture Detection (e.g., Needle driving)
Tracking Instruments and Spatial Context
Recognizing a high-level phase requires a precise understanding of the low-level objects within the frame. Models must classify, localize, and track surgical instruments in real time.
This task is exceptionally challenging because tools frequently disappear behind organs, become coated in blood, or create severe specular reflections. To bypass the need for tedious, frame-by-frame manual bounding box annotations, researchers use weakly supervised learning. For example, Hisham ElMoaqet et al. utilized a masked image modeling transformer (BEiT) in Using Masked Image Modelling Transformer Architecture for Laparoscopic Surgical Tool Classification and Localization. By training the network to predict missing parts of an image, the model naturally learns the structural geometry of surgical tools. The internal attention maps of the transformer can then be leveraged to localize the tools, achieving high spatial accuracy using only video-level labels.
For more complex robotic interventions, tracking extends beyond 2D bounding boxes to full 3-dimensional awareness. In Next-generation surgical navigation: Marker-less multi-view 6DoF pose estimation of surgical instruments, Jonas Hein et al. proved that marker-less, multi-view RGB-D video streams can yield 6-Degree-of-Freedom (6DoF) instrument tracking that rivals traditional hardware-based tracking systems. This works by fusing geometric depth data with multi-camera visual perspectives, ensuring that even if one camera's view is occluded by a surgeon's hand, the instrument's position is accurately maintained by the others.
Evaluating Skill and Detecting Errors
Once a system can reliably track what is happening and where, it becomes possible to evaluate how well the task is being performed. This shifts the paradigm from simple recognition to objective clinical assessment.
In Deep learning prediction of error and skill in robotic prostatectomy suturing, Nazir Sirajudeen et al. utilized the SAR-RARP50 dataset to train models capable of detecting frame-by-frame surgical gestures and errors. The intuition here is that expert surgeons move with higher kinematic efficiency and distinct, predictable patterns compared to trainees. By analyzing deviations from these optimal spatial-temporal trajectories, deep learning models can automatically flag technical errors (such as tissue trauma or slip-ups during suturing) and provide granular feedback for surgical training.
The Necessity of Specialized Benchmarks
General-purpose computer vision models fail when applied directly to the surgical domain. A model trained on everyday objects cannot handle the fluid dynamics of a sudden hemorrhage or the subtle visual distinction between different types of internal tissue.
Progress in this field is fundamentally driven by highly specialized, curated datasets that capture realistic clinical challenges:
- Cholec80 / CholecT45: Benchmarks for laparoscopic cholecystectomy, widely used for tool classification, localization, and phase recognition.
- HemoSet: Introduced by Michael C. Yip et al. in HemoSet: The First Blood Segmentation Dataset for Automation of Hemostasis Management, this dataset captures controlled hemorrhaging in live animal models to train algorithms specifically for automated blood detection and hemostasis management.
As these datasets grow, the field is expanding beyond specialized classification networks toward multimodal architectures. Esther Helene Stueker et al. explored this frontier in Vision-language models for automated video analysis and documentation in laparoscopic surgery: a proof-of-concept study. By evaluating frontier Vision-Language Models (VLMs) on surgical benchmarks, the authors showed that while these models can identify objects and broad phases via in-context learning, they still struggle with fine-grained clinical tasks like pathology grading. This underscores an ongoing reality in the field: achieving the level of precision required for clinical deployment demands domain-specific temporal modeling, rigorous expert annotation, and architectures designed to handle the unique constraints of surgical environments.
Key Foundational Work
Recognizing Surgical Activities with Recurrent Neural Networks (2016) — contributed 1 concept elements to this direction.
Quantification of Contrast Recognizability during Brilliant Blue G– and Indocyanine Green–Assisted Chromovitrectomy (2011) — contributed 1 concept elements to this direction.