A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms

Abdolrahim Kadkhodamohammadi, Afshin Gangi, Michel de Mathelin, Nicolas Padoy

Abstract

Many approaches have been proposed for human pose estimation in single and multi-view RGB images. However, some environments, such as the operating room, are still very challenging for state-of-the-art RGB methods. In this paper, we propose an approach for multi-view 3D human pose estimation from RGB-D images and demonstrate the benefits of using the additional depth channel for pose refinement beyond its use for the generation of improved features. The proposed method permits the joint detection and estimation of the poses without knowing a priori the number of persons present in the scene. We evaluate this approach on a novel multi-view RGB-D dataset acquired during live surgeries and annotated with ground truth 3D poses.

Introduction

Human Pose Estimation (HPE)—the ability for a computer to identify and track body parts—is a well-studied problem for many everyday scenarios. However, the Operating Room (OR) remains a significant challenge for computer vision. Standard RGB-based models often fail in this environment due to several factors:

  1. Visual Uniformity: Surgeons and nurses wear loose, textureless blue or green gowns that blend into each other and the background.
  2. Clutter and Occlusions: The room is packed with articulated arms, monitors, and multiple people constantly moving, leading to frequent "ghost" detections or hidden limbs.
  3. Restricted Placement: Cameras must stay out of the way of sterile fields and moving equipment, limiting optimal viewing angles.

Existing tools like the Kinect skeleton tracker often fail here because they rely on clean background subtraction, which is nearly impossible in a crowded, dynamic surgery.

Methods

The authors propose a three-stage pipeline to tackle these challenges by leveraging multi-view RGB-D data.

Multi-view RGB and Depth (RGB-D) data captured from three camera angles in a live operating room.
  1. Deep3DPS (Single-View Detector): The system extends a "3D Pictorial Structures" model. Instead of using handcrafted features, it uses a deep Convolutional Neural Network (ResNet-based) trained on RGB-D (Color + Depth) data to detect body parts. Crucially, it enforces 3D kinematic constraints directly in 3D space.
  2. Random Forest Room Prior: To filter out "spurious" detections (like a machine being mistaken for a person), a Random Forest classifier was trained. This model learns the "logic" of the OR—where people are likely to stand and what a valid human skeleton looks like in 3D—to reject impossible poses.
  3. Multi-view Optimization: The system merges detections from three different RGB-D cameras. It uses a novel energy function that considers part detection confidence, depth-based reprojection (ensuring the 3D pose matches the physical surfaces seen by the camera), and 3D joint distances. This is solved using the fast-PD algorithm to refine the final 3D coordinates.

Datasets

A core contribution of this work is the creation of a novel dataset:

  • Fine-tuning Dataset: A previously released single-view clinician dataset (1,451 frames) was used to adapt the deep network to the OR environment.
  • New Multi-view OR Dataset: Recorded over four days of live surgeries, this dataset includes synchronized RGB-D streams from three views, with 1,378 clinicians manually annotated in 3D across 741 multi-view frames. This represents a significant benchmark for the community.

Key Results and Insights

The proposed approach demonstrates that depth is more than just a feature; it allows the model to back-project 2D points into 3D space, enforcing physical body constraints that are more reliable than 2D pixel distances.

Qualitative results of the multi-view 3D human pose estimation system, showing skeletons overlaid on clinicians across different views.
  • Superior Accuracy: The "Deep3DPS" model achieved a 76.9% PCK (Probability of Correct Keypoint), vastly outperforming standard RGB models like Yang and Ramanan (25.2%) and previous single-view RGB-D methods (69.6%).
  • The Power of Multiple Views: Multi-view optimization significantly reduced localization errors. For example, the average 3D error for a wrist detection dropped from 32 cm (one view) to 18 cm (three views) after optimization.
  • Robustness: The Random Forest prior was over 84% accurate in identifying and removing "ghost" skeletons, ensuring the system only tracks real people.

Conclusion and Future Work

While the system shows great promise, some limitations remain. "Left vs. right" labeling in individual views is still somewhat unreliable, though the multi-view voting system helps mitigate this. Additionally, the study focused on upper-body poses because the lower body is almost always occluded by operating tables and equipment. Future work may address these issues and further optimize the computational complexity for real-time surgical assistance.

Summary updated

Abstract

Many approaches have been proposed for human pose estimation in single and multi-view RGB images. However, some environments, such as the operating room, are still very challenging for state-of-the-art RGB methods. In this paper, we propose an approach for multi-view 3D human pose estimation from RGB-D images and demonstrate the benefits of using the additional depth channel for pose refinement beyond its use for the generation of improved features. The proposed method permits the joint detection and estimation of the poses without knowing a priori the number of persons present in the scene. We evaluate this approach on a novel multi-view RGB-D dataset acquired during live surgeries and annotated with ground truth 3D poses.
BibTeX
@inproceedings{kadkhodamohammadi2017multi,
  author = {Abdolrahim Kadkhodamohammadi and Afshin Gangi and Michel de Mathelin and Nicolas Padoy},
  title = {{A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2017},
  url = {https://openreview.net/forum?id=kHPl0BGrvD},
}
Figure