Putting the pieces together: Connected Poselets for human pose estimation

Brian Holt, Eng-Jon Ong, Helen Cooper, Richard BowdenWorkshops @ ICCV 2011 · OpenReview

Abstract

We propose a novel hybrid approach to static pose estimation called Connected Poselets. This representation combines the best aspects of part-based and example-based estimation. First detecting poselets extracted from the training data; our method then applies a modified Random Decision Forest to identify Poselet activations. By combining keypoint predictions from poselet activations within a graphical model, we can infer the marginal distribution over each keypoint without any kinematic constraints. Our approach is demonstrated on a new publicly available dataset with promising results.

Introduction

Human pose estimation is a fundamental challenge in computer vision, critical for applications ranging from gesture recognition to human-computer interaction. The human body's high degree of deformability, coupled with variations in clothing, body shape, and camera angles, makes accurate estimation particularly difficult. While systems like the original Microsoft Kinect achieved success using depth sensors, they relied on massive datasets—nearly a million frames—to generalize effectively.

This paper introduces "Connected Poselets," a hybrid approach designed to achieve robust pose estimation from depth images even with limited training data. By bridging the gap between part-based and example-based methods, the proposed system can learn complex poses from just a few hundred images, specifically targeting scenarios like upper-body tracking in domestic environments.

Methodology

The Connected Poselets framework operates through a two-stage pipeline that transitions from local feature detection to global pose inference.

Poselet Detection

A "poselet" is defined as a visually and geometrically consistent configuration of body parts, such as a specific orientation of the head and shoulders. Unlike traditional methods that might use HOG features, this approach utilizes raw depth pixel intensities.

Examples of depth-based poselets used for feature extraction.

The detection process employs a modified Random Decision Forest (RDF). Standard Classification and Regression Trees (CART) typically split data based on a single pixel value. In contrast, this method uses a "linear combination split" that compares two pixels ($X_l \le X_m$). This technique is more effective at capturing relative depth relationships, allowing the model to recognize body shapes regardless of their absolute distance from the sensor.

Hierarchical Inference

Once poselets are detected, they act as "votes" for the locations of specific joints (keypoints). To refine these predictions and ensure anatomical plausibility, the authors employ a hierarchical graphical model.

The Connected Poselets pipeline: from scanning windows to joint proposals.

The model follows the body's natural hierarchy:

  1. Level 1 (Root): The torso, including the head, neck, shoulders, and hips.
  2. Level 2: Elbows, which are conditionally dependent on the shoulders.
  3. Level 3: Hands, which are conditionally dependent on the elbows.

By treating the torso as the high-confidence root, the system uses its position to constrain the search space for more mobile and "jittery" parts like the hands and elbows. This Bayesian approach provides a strong prior without enforcing rigid kinematic constraints.

Experimental Setup and Datasets

To evaluate the system, the authors introduced the CDC4CV Poselets dataset. This dataset consists of depth images and annotations of three participants performing various motions. Notably, it contains only 345 training images and 347 test images. This scale is significantly smaller than the "big data" requirements of commercial systems but is comparable to academic benchmarks like the Buffy or ETHZ Stickmen datasets, highlighting the model's ability to generalize from limited data.

Results and Discussion

The system's performance was measured using the Percentage of Correctly Matched Parts (PCP) metric.

  • Accuracy and Inference: The overall system achieved 67% accuracy. The inclusion of the hierarchical graphical model was crucial, improving total accuracy from 61% to 67%.
  • Joint-Specific Performance: The system excels at identifying central body parts, with 99% accuracy for the head and 93% for the torso. However, accuracy declines for extremities, with forearms and hands reaching only 22-33%. This disparity underscores the difficulty of tracking small, fast-moving parts compared to the central body mass.

Key Insights

The hybrid nature of Connected Poselets allows the model to capture more context than isolated joint detectors while maintaining greater flexibility than whole-body templates. Furthermore, the use of relative depth comparisons in the RDF nodes proves to be a robust feature for depth-based pose estimation.

Conclusion and Future Work

While Connected Poselets demonstrate strong performance on central body parts with minimal training data, the "drift" in accuracy toward the extremities remains a challenge. Future research directions include integrating this static estimator into a temporal tracking framework. By leveraging motion data across video frames, the system could use temporal consistency to better constrain the positions of hands and forearms, potentially overcoming the limitations of purely static estimation.

Summary updated

Abstract

We propose a novel hybrid approach to static pose estimation called Connected Poselets. This representation combines the best aspects of part-based and example-based estimation. First detecting poselets extracted from the training data; our method then applies a modified Random Decision Forest to identify Poselet activations. By combining keypoint predictions from poselet activitions within a graphical model, we can infer the marginal distribution over each keypoint without any kinematic constraints. Our approach is demonstrated on a new publicly available dataset with promising results.
BibTeX
@inproceedings{holt2011putting,
  author = {Brian Holt and Eng-Jon Ong and Helen Cooper and Richard Bowden},
  title = {{Putting the pieces together: Connected Poselets for human pose estimation}},
  booktitle = {Workshops @ ICCV},
  year = {2011},
  url = {https://openreview.net/forum?id=ATMt8tLtM7},
}
Figure