TL;DR
This paper introduces a novel multi-target multi-camera tracking (MTMCT) system to automatically recognize postoperative patient phases, track their location, and generate timestamps. By extending context awareness beyond the operating room, this work lays the groundwork for real-time clinical support, improved documentation, and enhanced patient care.
What Problem Does This Paper Address?
While deep learning has made significant strides in understanding surgical procedures within the operating room (OR), the critical postoperative phase remains largely underexplored. This period, from the end of surgery to recovery or intensive care, is crucial for patient outcomes but lacks intelligent assistance and automated documentation. Current systems often stop at the OR door, leaving a gap in continuous patient monitoring and workflow optimization. This paper aims to bridge this gap by developing a system that can automatically track patients and their progress through postoperative workflows, providing valuable context for clinical staff and reducing their documentation burden.
Methods
The core of this work is a custom Multi-Target Multi-Camera Tracking (MTMCT) architecture. This system leverages the YOLOv8 object detection model and the ByteTrack algorithm for initial detection and tracking of key objects like OR tables, care beds, and patients from multiple camera feeds.
What's novel is the integration of medical domain knowledge into a "postoperative patient state module." This module comprises several components:
- Patient Checker: Addresses the challenge of tracking patients who are often partially obscured or wearing standardized attire by focusing on the overlap between patient and bed bounding boxes.
- Match Candidate Finder & Match Approver: These components are crucial for handling multi-camera data. They identify and validate potential matches between partial tracks from different cameras, using a novel approach called AFLink. AFLink relies on trajectory data in 3D space rather than appearance information, making it faster and more suitable for this application.
- State Machine (SM): This component defines the postoperative phases (Transport, Patient Transfer, Recovery) and uses the approved matches to transition between these states, automatically generating timestamps when a patient enters a new area.
- Match Conflict Solver: This ensures that tracks are not assigned to multiple patients, resolving ambiguities by considering patient states and, if necessary, similarity scores from AFLink.
Datasets
The authors created a new dataset at TUM University Hospital consisting of 19 reenacted postoperative patient flows. These flows cover three scenarios: patients moved to a transfer unit then to a recovery bed (S1), direct transfer to a recovery bed (S2), and transfer to an ICU bed (S3). To address the limitation of single-patient scenarios in the real data, they synthetically generated 252 additional patient flows to simulate multi-patient scenarios. The dataset includes bounding box annotations for OR tables, care beds, and patients, with videos recorded at 1080p resolution and 50 fps, and annotations at 25 fps.
Key Insights and Ideas
- Extending Context Beyond the OR: The paper highlights the significant gap in intelligent assistance for the postoperative phase, emphasizing its importance for patient outcomes.
- Domain-Specific Tracking: Reusing existing multi-object tracking solutions for traffic or pedestrians is insufficient. Integrating medical domain knowledge, particularly around patient and bed interactions, is key to success.
- AFLink for Robust Association: The use of AFLink, which relies on trajectory data rather than appearance, proves effective for associating tracks across multiple cameras in the complex postoperative environment, especially in multi-patient scenarios.
- Automated Documentation Potential: The system's ability to automatically generate timestamps and identify patient phases offers a powerful tool for reducing the manual documentation burden on clinical staff.
Key Results
The proposed MTMCT architecture demonstrates promising results:
- Postoperative Phase Recognition: Achieved an accuracy of 84.9 ± 5.9% in multi-patient scenarios.
- Timestamp Generation: Generated timestamps with an accuracy of 91.4 ± 1.5% (with a 1-second tolerance).
- Patient Tracking: Achieved a robust IDF1 score of 92.0 ± 3.6% for patient tracking consistency.
- Object Detection & Tracking: YOLOv8 showed good mAP, though patient detection was more challenging, especially in hallways and recovery units. ByteTrack performed well for OR tables and care beds but struggled more with patient tracking in certain areas.
Limitations and Future Work
The authors acknowledge several limitations:
- Dataset Size: The real-world dataset is small (19 flows), which could limit the generalizability of the model to different patient appearances or clinical setups.
- Patient Tracking Performance: While acceptable, patient tracking performance was weaker than for beds and OR tables, particularly in occluded or dynamic environments.
- Generalizability: The transferability to other clinical setups needs further investigation.
Future work suggested includes:
- Dataset Expansion: Recording more multi-patient scenarios and incorporating a wider variety of patient appearances and clinical environments.
- Improved Tracking: Exploring alternative object detection algorithms or segmentation methods to enhance patient tracking reliability.
- Advanced Similarity Metrics: Investigating additional similarity metrics for partial track matching.
- Ethical Considerations: Addressing ethical implications by focusing on processing live streams and storing only outputs (timestamps, phases, locations) to protect patient privacy.
Summary updated