Abstract
We consider the problem of parsing human poses and recognizing their actions in static images with part-based models. Most previous work in part-based models only considers rigid parts (e.g., torso, head, half limbs) guided by human anatomy. We argue that this representation of parts is not necessarily appropriate. In this paper, we introduce hierarchical poselets--a new representation for modeling the pose configuration of human bodies. Hierarchical poselets can be rigid parts, but they can also be parts that cover large portions of human bodies (e.g., torso + left arm). In the extreme case, they can be the whole bodies. The hierarchical poselets are organized in a hierarchical way via a structured model. Human parsing can be achieved by inferring the optimal labeling of this hierarchical model. The pose information captured by this hierarchical model can also be used as a intermediate representation for other high-level tasks. We demonstrate it in action recognition from static images.
Introduction
Traditional human parsing models often rely on "pictorial structures," representing the body as a collection of rigid, anatomically-defined parts. However, detecting small, isolated segments like a forearm is notoriously difficult due to background clutter and lack of distinctive features. This paper introduces hierarchical poselets, a multi-scale representation that captures visual patterns spanning multiple anatomical parts. By moving beyond simple rigid segments to include larger configurations (e.g., "torso + arm") and the whole body, the model provides a more robust framework for both pose estimation and action recognition.
Methods
The core innovation is the hierarchical poselet representation, which organizes parts into a structured model of varying granularity.
Hierarchical Representation
Instead of a flat set of limbs, the authors define 20 parts across different scales. These range from "primitive parts" (like the head) to "large parts" (combinations like torso + left arm) and finally the "whole body." This hierarchy is modeled as an undirected graphical model where the score for a specific pose configuration is determined by local appearance matching and spatial geometric priors.
Feature Engineering and Learning
The model utilizes Histogram of Oriented Gradients (HOG) descriptors. Crucially, the "cell size" for these features varies based on the part's scale: large parts use coarse features (12x12 pixels) to capture global shape, while small parts use fine-grained features (2x2 pixels) for detail.
Inference is performed using max-product belief propagation. For learning, a Max-Margin Structured SVM framework is used for parsing, while a Latent SVM is employed for action recognition, treating the underlying pose as a hidden variable.
Experimental Results
The model was evaluated across several benchmarks, including the UIUC People Dataset, a new Sport Image Dataset (1,299 images), the Still Image Action Dataset, and the Leeds Sport Data Set.
Human Parsing and Detection
On the UIUC dataset, the hierarchical model achieved a torso localization accuracy of 86.6%, significantly outperforming standard pictorial structures (70.9%). Interestingly, the "whole body" node in the hierarchy acts as an effective person detector, achieving 66.8% accuracy, which surpasses the state-of-the-art Felzenszwalb et al. (DPM) detector (48.58%) on this data.
Action Recognition
For action recognition on the Still Image dataset, the approach reached 65.15% overall accuracy, outperforming baseline SVMs (60.32%) and previous latent pose models (63.49%).
Discussion and Conclusion
A key insight of this work is that "context is king." While a forearm is hard to find, a "torso + arm" is much more distinctive and provides a strong spatial anchor for smaller segments. This approach results in poses that, even when incorrect, typically maintain a plausible human configuration rather than the "broken" results often seen in kinematic models.
Future work could involve automatically learning the part hierarchy from data rather than manual definition, potentially extending the model's applicability to non-human objects with complex structures.
Summary updated