Abstract: Human pose estimation in the operating room (OR) can benefit many applications, such as surgical activity recognition, radiation exposure monitoring and performance assessment. However, the OR is a very challenging environment for computer vision systems due to limited camera positioning possibilities, severe illumination changes and similar colors of clothes and equipments. This paper tackles the problem of human pose estimation in the OR using RGB-D images, hypothesizing that the combination of depth and color information will improve the pose estimation results in such a difficult environment. We propose an approach based on pictorial structures that makes use of both channels of the RGB-D camera and also introduce a new feature descriptor for depth images, called histogram of depth differences (HDD), that captures local depth level changes. To quantitatively evaluate the proposed approach, we generate a novel dataset by manually annotating images recorded from different camera views during several days of live surgeries. Our experiments show that the pictorial structures (PS) approach applied on depth images using HDD outperforms the state-of-the art PS approach applied on the corresponding color images by over 11%. Furthermore, the proposed HDD descriptor has superior performance when compared to two other classical descriptors applied on depth images. Finally, the appearance models generated from the depth images perform better than those generated from the color images, and the combination of both improves the overall results. We therefore conclude that it is highly beneficial to use depth information in the pictorial structure model and also for human pose estimation in operating rooms.
The Challenge of the Operating Room
Human Pose Estimation (HPE)—the ability for a computer to identify the location of joints like elbows, wrists, and shoulders—is a foundational step for higher-level tasks like monitoring radiation exposure or assessing surgical performance. However, the Operating Room is a "computer vision nightmare" for several reasons:
- Visual Monotony: Surgeons, nurses, and equipment are often covered in the same shades of blue or green, making it nearly impossible for color-based algorithms to distinguish a person's arm from the background.
- Lighting Volatility: Intense surgical lamps and changing ambient light create harsh shadows and highlights that confuse standard cameras.
- Physical Constraints: Cameras cannot be placed anywhere; they must be mounted out of the way of articulated ceiling arms and sterile zones, leading to awkward viewing angles.
While depth sensors (like the Kinect) help, existing algorithms often require a clean "foreground mask" (knowing exactly where the person is vs. the background), which is nearly impossible to get in a crowded, busy OR.
Methodology: Pictorial Structures and HDD
The researchers built their approach on the Pictorial Structures (PS) framework, specifically the "Flexible Mixtures of Parts" model. This model treats the human body like a collection of parts connected by "springs" that represent anatomical constraints (e.g., your wrist is always near your elbow).
The core innovation is the Histogram of Depth Differences (HDD) descriptor:
- The Concept: Instead of looking at colors, HDD looks at local changes in depth. It uses four specific kernels (mathematical filters) to capture how the "surface level" of a scene changes.
- Depth Invariance: The responses are normalized by the distance from the camera, meaning a person's arm looks the same to the algorithm whether they are 2 meters or 4 meters away.
- Robustness: By quantizing these depth changes into a 3D histogram, the descriptor becomes resistant to the "noise" often found in raw depth data.
The authors compared this new HDD feature against standard color features (I-HOG), depth-based HOG (D-HOG), and surface normal vectors (HONV).
A Novel RGB-D Dataset
To evaluate their approach, the authors introduced a novel RGB-D dataset recorded during live surgeries. Unlike previous datasets recorded in empty "mock" ORs, this one captures the actual chaos of live operations. It was recorded over seven half-days using an Asus Xtion Pro Live camera and contains 1,451 frames with 1,991 manually annotated persons, including surgeons, nurses, and anesthesiologists wearing various clinical gear.
Results and Discussion
The performance was measured using PCK (Probability of Correct Keypoint), which checks if the predicted joint is within a certain distance of the actual joint. The results demonstrated that depth information is a far more reliable signal than color in the OR environment.
- Color Only (Baseline): 63.25% accuracy.
- Depth Only (HDD): 74.69% accuracy (an 11.44% improvement over color).
- Combined (Color + HDD): 76.64% accuracy.
The researchers found that depth descriptors were particularly good at finding shoulders, likely due to the sharp depth "step" between a person and the background. The combination of color and depth provided the best results, proving that while depth is the "heavy lifter," color still provides useful complementary details, especially for difficult joints like elbows, wrists, and hips.
Conclusion and Future Work
This study concludes that depth information is highly beneficial for human pose estimation in operating rooms. While the model still struggles with severe occlusions—such as when a surgeon leans deeply over a patient—the introduction of HDD provides a significant leap in accuracy. Future work may involve moving from 2D image space to full 3D inference and incorporating temporal logic to track joints across video frames.
Summary updated