Abstract
Reliable human pose estimation (HPE) is essential to many clinical applications, such as surgical workflow analysis, radiation safety monitoring and human-robot cooperation. Proposed methods for the operating room (OR) rely either on foreground estimation using a multi-camera system, which is a challenge in real ORs due to color similarities and frequent illumination changes, or on wearable sensors or markers, which are invasive and therefore difficult to introduce in the room. Instead, we propose a novel approach based on Pictorial Structures (PS) and on RGB-D data, which can be easily deployed in real ORs. We extend the PS framework in two ways. First, we build robust and discriminative part detectors using both color and depth images. We also present a novel descriptor for depth images, called histogram of depth differences (HDD). Second, we extend PS to 3D by proposing 3D pairwise constraints and a new method that makes exact inference tractable. Our approach is evaluated for pose estimation and clinician detection on a challenging RGB-D dataset recorded in a busy operating room during live surgeries. We conduct series of experiments to study the different part detectors in conjunction with the various 2D or 3D pairwise constraints. Our comparisons demonstrate that 3D PS with RGB-D part detectors significantly improves the results in a visually challenging operating environment.
Introduction: The Challenge of the Operating Room
Estimating human pose in an operating room (OR) is a significant challenge for standard computer vision. ORs are cluttered with equipment, lighting changes constantly during surgery, and—most critically—staff wear nearly identical medical scrubs. This "sea of blue/green" makes it difficult to distinguish individuals or body parts based on color alone.
Existing solutions typically fall into two categories: invasive methods using wearable sensors or markers, and unreliable vision systems that fail when foreground subtraction is compromised. This research proposes a non-invasive, vision-based approach that leverages depth data to handle the visual complexity of live surgical environments.
Methodology: Extending Pictorial Structures to 3D
The researchers built upon the Pictorial Structures (PS) framework, which represents the human body as a collection of parts (nodes) connected by flexible joints (edges). They extended this framework in three key ways:
- Multimodal Appearance Models: Beyond standard color (HOG descriptors), the model incorporates depth information. A new descriptor, the Histogram of Depth Differences (HDD), was introduced. HDD is "depth-invariant," focusing on local surface changes rather than absolute distance, making it robust as subjects move relative to the camera.
- 3D Pairwise Constraints: Standard 2D PS can erroneously "stitch together" parts from different people if they overlap in a 2D image. The authors proposed a 3D deformation model using Euclidean distances to ensure body parts are only connected if they are physically adjacent in 3D space.
- Efficient 3D Inference: To manage the computational complexity of 3D math, the authors developed a state-space pruning algorithm. This uses 3D information to quickly discard unlikely part locations, making exact 3D inference fast enough for practical application.
Dataset and Evaluation
The approach was tested on a new RGB-D dataset captured during live surgeries using an Asus Xtion Pro Live camera. The dataset includes 1,451 annotated frames from seven half-days of surgery, featuring 3,023 bounding boxes for staff and 1,991 detailed pose annotations across nine upper-body parts. The data captures severe occlusions and clutter typical of a busy OR.
Results: The Power of Depth and 3D Constraints
Performance was evaluated using the Probability of Correct Keypoint (PCK) metric. The results highlight the superiority of the multimodal 3D approach:
- Baseline (Color-only 2D): 63.3% accuracy.
- Proposed (RGB-D + 3D Constraints): 81.5% accuracy.
- HDD Performance: The specialized HDD descriptor alone improved the baseline by approximately 11%.
- Detection: For clinician detection (bounding boxes), the 3D approach achieved an Average Precision (AP) of 89.7%, significantly outperforming the Deformable Part Models (DPM) baseline.
The experiments confirmed that in environments where color is a weak signal, the shape information provided by depth is far more reliable. Furthermore, a hybrid constraint combining 2D pixel displacement with 3D distances proved most effective, as it mitigates the noise inherent in consumer-grade RGB-D sensors.
Conclusion and Future Work
This study demonstrates that extending Pictorial Structures with 3D depth data significantly improves human pose estimation in challenging clinical settings. While the single-viewpoint system is effective, it still faces challenges with self-occlusion (e.g., a surgeon leaning over a patient). Future work will focus on extending this framework to multi-view systems, fusing data from several cameras to provide a 360-degree view of the surgical team and further minimize the impact of occlusions.
Summary updated