3D Pictorial Structures on RGB-D Data for Articulated Human Detection in Operating Rooms

Abdolrahim Kadkhodamohammadi, Afshin Gangi, Michel de Mathelin, Nicolas Padoy

Abstract

Reliable human pose estimation (HPE) is essential to many clinical applications, such as surgical workflow analysis, radiation safety monitoring and human-robot cooperation. Proposed methods for the operating room (OR) rely either on foreground estimation using a multi-camera system, which is a challenge in real ORs due to color similarities and frequent illumination changes, or on wearable sensors or markers, which are invasive and therefore difficult to introduce in the room. Instead, we propose a novel approach based on Pictorial Structures (PS) and on RGB-D data, which can be easily deployed in real ORs. We extend the PS framework in two ways. First, we build robust and discriminative part detectors using both color and depth images. We also present a novel descriptor for depth images, called histogram of depth differences (HDD). Second, we extend PS to 3D by proposing 3D pairwise constraints and a new method that makes exact inference tractable. Our approach is evaluated for pose estimation and clinician detection on a challenging RGB-D dataset recorded in a busy operating room during live surgeries. We conduct series of experiments to study the different part detectors in conjunction with the various 2D or 3D pairwise constraints. Our comparisons demonstrate that 3D PS with RGB-D part detectors significantly improves the results in a visually challenging operating environment.

Introduction: The Challenge of the Operating Room

Estimating human pose in an operating room (OR) is a significant challenge for standard computer vision. ORs are cluttered with equipment, lighting changes constantly during surgery, and—most critically—staff wear nearly identical medical scrubs. This "sea of blue/green" makes it difficult to distinguish individuals or body parts based on color alone.

A typical operating room scene illustrating the visual challenges of color-based detection due to uniform surgical attire.

Existing solutions typically fall into two categories: invasive methods using wearable sensors or markers, and unreliable vision systems that fail when foreground subtraction is compromised. This research proposes a non-invasive, vision-based approach that leverages depth data to handle the visual complexity of live surgical environments.

Methodology: Extending Pictorial Structures to 3D

The researchers built upon the Pictorial Structures (PS) framework, which represents the human body as a collection of parts (nodes) connected by flexible joints (edges). They extended this framework in three key ways:

  1. Multimodal Appearance Models: Beyond standard color (HOG descriptors), the model incorporates depth information. A new descriptor, the Histogram of Depth Differences (HDD), was introduced. HDD is "depth-invariant," focusing on local surface changes rather than absolute distance, making it robust as subjects move relative to the camera.
  2. 3D Pairwise Constraints: Standard 2D PS can erroneously "stitch together" parts from different people if they overlap in a 2D image. The authors proposed a 3D deformation model using Euclidean distances to ensure body parts are only connected if they are physically adjacent in 3D space.
  3. Efficient 3D Inference: To manage the computational complexity of 3D math, the authors developed a state-space pruning algorithm. This uses 3D information to quickly discard unlikely part locations, making exact 3D inference fast enough for practical application.
Visualization of the Pictorial Structure framework, where body parts are modeled as nodes connected by kinematic constraints.

Dataset and Evaluation

The approach was tested on a new RGB-D dataset captured during live surgeries using an Asus Xtion Pro Live camera. The dataset includes 1,451 annotated frames from seven half-days of surgery, featuring 3,023 bounding boxes for staff and 1,991 detailed pose annotations across nine upper-body parts. The data captures severe occlusions and clutter typical of a busy OR.

Results: The Power of Depth and 3D Constraints

Performance was evaluated using the Probability of Correct Keypoint (PCK) metric. The results highlight the superiority of the multimodal 3D approach:

  • Baseline (Color-only 2D): 63.3% accuracy.
  • Proposed (RGB-D + 3D Constraints): 81.5% accuracy.
  • HDD Performance: The specialized HDD descriptor alone improved the baseline by approximately 11%.
  • Detection: For clinician detection (bounding boxes), the 3D approach achieved an Average Precision (AP) of 89.7%, significantly outperforming the Deformable Part Models (DPM) baseline.

The experiments confirmed that in environments where color is a weak signal, the shape information provided by depth is far more reliable. Furthermore, a hybrid constraint combining 2D pixel displacement with 3D distances proved most effective, as it mitigates the noise inherent in consumer-grade RGB-D sensors.

Conclusion and Future Work

This study demonstrates that extending Pictorial Structures with 3D depth data significantly improves human pose estimation in challenging clinical settings. While the single-viewpoint system is effective, it still faces challenges with self-occlusion (e.g., a surgeon leaning over a patient). Future work will focus on extending this framework to multi-view systems, fusing data from several cameras to provide a 360-degree view of the surgical team and further minimize the impact of occlusions.

Summary updated

Abstract

Reliable human pose estimation (HPE) is essential to many clinical applications, such as surgical workflow analysis, radiation safety monitoring and human-robot cooperation. Proposed methods for the operating room (OR) rely either on foreground estimation using a multi-camera system, which is a challenge in real ORs due to color similarities and frequent illumination changes, or on wearable sensors or markers, which are invasive and therefore difficult to introduce in the room. Instead, we propose a novel approach based on Pictorial Structures (PS) and on RGB-D data, which can be easily deployed in real ORs. We extend the PS framework in two ways. First, we build robust and discriminative part detectors using both color and depth images. We also present a novel descriptor for depth images, called histogram of depth differences (HDD). Second, we extend PS to 3D by proposing 3D pairwise constraints and a new method that makes exact inference tractable. Our approach is evaluated for pose estimation and clinician detection on a challenging RGB-D dataset recorded in a busy operating room during live surgeries. We conduct series of experiments to study the different part detectors in conjunction with the various 2D or 3D pairwise constraints. Our comparisons demonstrate that 3D PS with RGB-D part detectors significantly improves the results in a visually challenging operating environment.
BibTeX
@article{kadkhodamohammadi2017d,
  author = {Abdolrahim Kadkhodamohammadi and Afshin Gangi and Michel de Mathelin and Nicolas Padoy},
  title = {{3D Pictorial Structures on RGB-D Data for Articulated Human Detection in Operating Rooms}},
  journal = {Medical Image Analysis},
  year = {2017},
  url = {https://openreview.net/forum?id=b0OALCyTbJ},
}
Figure