Action recognition in medical environments for robotic assistance

Sonja Stabenow, L Wagner, Alois Knoll, Klaus Bengler, Dirk WilhelmInternational Journal of Computer Assisted Radiology and Surgery 2025 · DOI

Abstract

The handover phases in two medical contexts, a minimally invasive surgery and a wound dressing on the patient station, are recognized with the proposed framework. This lays a first step for the integration of robotic assistance in the handover of medical material or instruments.

Introduction

In high-stakes medical environments, seamless collaboration is often driven by implicit cues rather than explicit instructions. A scrub nurse, for example, anticipates a surgeon's need for a scalpel by observing their movements and the procedure's progress. To integrate robotic assistants into these teams and alleviate staff workload, robots must be capable of "understanding" these unspoken transitions. However, medical settings present unique challenges: identical baggy scrubs hide body contours, lighting is often dim, and workspaces are crowded. This paper investigates whether modern skeleton-based Human Action Recognition (HAR) algorithms can reliably identify handover phases across two distinct clinical settings: the Operating Room (OR) and a Patient Ward.

Methodology

The researchers proposed a multi-stage pipeline to recognize actions from video data while maintaining privacy and handling visual uniformity.

Overview of the multi-stage pipeline for medical human action recognition, illustrating the flow from input frames to pose estimation, feature extraction, and final action classification.
  1. Pose Estimation: Algorithms like Swin Transformer and RTMpose3D extract 2D and 3D "skeletons." Skeletons abstract away individual appearances and are computationally efficient.
  2. Feature Extraction: Data is processed into three streams: Joints (2D coordinates), Bones (anatomical connections), and Motion (positional changes over time).
  3. Action Recognition: Three state-of-the-art models were compared: ST-GCN (Spatiotemporal Graph Convolutional Network), CTR-GCN (which dynamically refines skeleton topology), and SkateFormer (a transformer-based approach).
  4. Anticipation: The models were also tested on their ability to predict actions 0.5 to 2.0 seconds before they occur.

Datasets

The study utilized three primary data sources:

  • Ward Dataset: 20 recreated wound dressing scenes on a dummy, involving 10 participants (~31k frames). Actions include approach, reach, transfer, retract, and at patient.
  • OR Dataset: 14 real surgical procedures (cholecystectomies and sigmoid resections) totaling ~344k frames. Actions include instrument extraction, handover, instrument insertion, operate, and idle.
  • MM-OR: A public dataset of knee arthroplasties used to evaluate cross-domain generalization.

Key Results and Insights

  • Environment Performance: The SkateFormer model achieved an F1 score of 0.941 in the Ward setting, where movements are expansive. In the more complex OR environment, ST-GCN performed best with an F1 score of 0.736, reflecting the difficulty of detecting subtle movements often obscured by the surgical site.
  • Action Anticipation: Models remained robust when predicting the future. In the Ward, the F1 score only dropped slightly (from 0.94 to 0.91) when predicting 2 seconds ahead.
  • Generalization: A model trained on abdominal surgeries achieved an F1 score of 0.658 on knee surgery data without fine-tuning, suggesting a consistent "grammar" for medical handovers.
  • Privacy and Uniformity: Skeleton-based models successfully ignore the visual similarities of scrubs while preserving the privacy of staff and patients.

Conclusion

This research demonstrates that skeleton-based HAR can effectively detect and even anticipate critical handover events in medical settings. While the OR remains a challenging environment due to class imbalance and subtle motions, the results provide a foundation for future robotic assistants. Future work will focus on incorporating multimodal data (e.g., audio or laparoscopic video) and developing real-time assistive loops for robotic hardware.

Summary updated

Abstract

The handover phases in two medical contexts, a minimally invasive surgery and a wound dressing on the patient station, are recognized with the proposed framework. This lays a first step for the integration of robotic assistance in the handover of medical material or instruments.
BibTeX
@article{stabenow2025action,
  author = {Sonja Stabenow and L Wagner and Alois Knoll and Klaus Bengler and Dirk Wilhelm},
  title = {{Action recognition in medical environments for robotic assistance}},
  journal = {International Journal of Computer Assisted Radiology and Surgery},
  year = {2025},
  doi = {10.1007/s11548-025-03551-6},
}
Figure