Attribute And-Or Grammar for Joint Parsing of Human Attributes, Part and Pose

Seyoung Park, Bruce Xiaohan Nie, Song-Chun Zhu

Abstract

This paper presents an attribute and-or grammar (A-AOG) model for jointly inferring human body pose and human attributes in a parse graph with attributes augmented to nodes in the hierarchical representation. In contrast to other popular methods in the current literature that train separate classifiers for poses and individual attributes, our method explicitly represents the decomposition and articulation of body parts, and accounts for the correlations between poses and attributes. The A-AOG model is an amalgamation of three traditional grammar formulations: (i) Phrase structure grammar representing the hierarchical decomposition of the human body from whole to parts; (ii) Dependency grammar modeling the geometric articulation by a kinematic graph of the body pose; and (iii) Attribute grammar accounting for the compatibility relations between different parts in the hierarchy so that their appearances follow a consistent style. The parse graph outputs human detection, pose estimation, and attribute prediction simultaneously, which are intuitive and interpretable.

Introduction

Traditionally, computer vision treats pose estimation (locating joints) and attribute prediction (identifying characteristics like gender or clothing) as independent tasks. This siloed approach often leads to error propagation—where a poorly localized limb causes an attribute classifier to fail—and a lack of contextual reasoning. This research, published in IEEE Transactions on Pattern Analysis and Machine Intelligence, proposes a unified framework that leverages the deep correlations between a person's physical structure and their attributes.

The Attribute And-Or Grammar (A-AOG) Framework

The core innovation of this work is the Attribute And-Or Grammar (A-AOG). This hierarchical model treats human understanding as a parsing problem, combining three distinct grammar types to ensure structural and semantic consistency:

  1. Phrase Structure Grammar: Handles the hierarchical "whole-to-parts" decomposition (e.g., full body to upper/lower body to limbs).
  2. Dependency Grammar: Models the kinematic relationships, ensuring that joints and parts articulate in anatomically plausible ways.
  3. Attribute Grammar: Manages compatibility across the hierarchy, ensuring that attributes like gender or age are consistent across all detected body parts.
The A-AOG framework integrates body parsing, pose estimation, and attribute parsing into a single unified representation.

The system operates by first generating "part proposals" using a Convolutional Neural Network (CNN) based on Faster R-CNN. These proposals, which include potential locations and attribute likelihoods, are then processed by the A-AOG. Using a greedy beam-search algorithm, the model finds the most probable "parse graph"—a complete representation of the person's pose and attributes that satisfies all grammatical constraints.

Key Insights: Attributes as Structural Constraints

A major finding of this research is that attributes can serve as powerful global constraints to resolve pose ambiguity. In crowded scenes or under occlusion, a standard pose estimator might confuse the limbs of two adjacent people. By enforcing attribute consistency (e.g., ensuring all parts of a person match a specific clothing color or gender), the A-AOG can correctly reject parts that belong to others.

Comparison of pose estimation with and without attribute constraints. The constraints help resolve ambiguities in complex scenes.

Furthermore, the model uses mutual information to statistically link specific attributes to relevant nodes in the hierarchy. For instance, "long pants" is strongly correlated with the lower body and leg nodes, while "gender" acts as a global attribute influencing the entire parse graph. This structured approach provides an explainable output; the resulting parse graph explicitly shows the reasoning behind each prediction, moving away from "black-box" neural network outputs.

Experimental Results

The researchers evaluated the A-AOG on the "Attributes of People" dataset and a new "Pedestrian Attribute Dataset." The results demonstrated the clear advantage of joint modeling:

  • Pose Estimation: The joint model achieved a 52.4% Average Percentage of Correct Parts (PCP), a significant improvement over the 34.5% baseline that ignored attribute constraints.
  • Attribute Prediction: The model reached a mean Average Precision (mAP) of 91.6%, outperforming state-of-the-art methods like PANDA and R*CNN.
  • Mutual Benefit: Diagnostic tests showed that pose estimation improved attribute classification by approximately 2.7%, while attributes improved pose estimation by 4.6%.

Conclusion

The A-AOG framework successfully demonstrates that pose and attributes are not independent signals but are deeply intertwined. While the model currently requires strong supervision and faces computational challenges in extremely dense crowds, it sets a new standard for interpretable and accurate human parsing. Future research will focus on reducing annotation requirements and optimizing the inference process for even more complex scenarios.

Summary updated

Abstract

This paper presents an attribute and-or grammar (A-AOG) model for jointly inferring human body pose and human attributes in a parse graph with attributes augmented to nodes in the hierarchical representation. In contrast to other popular methods in the current literature that train separate classifiers for poses and individual attributes, our method explicitly represents the decomposition and articulation of body parts, and account for the correlations between poses and attributes. The A-AOG model is an amalgamation of three traditional grammar formulations: (i) Phrase structure grammar representing the hierarchical decomposition of the human body from whole to parts; (ii) Dependency grammar modeling the geometric articulation by a kinematic graph of the body pose; and (iii) Attribute grammar accounting for the compatibility relations between different parts in the hierarchy so that their appearances follow a consistent style. The parse graph outputs human detection, pose estimation, and attribute prediction simultaneously, which are intuitive and interpretable. We conduct experiments on two tasks on two datasets, and experimental results demonstrate the advantage of joint modeling in comparison with computing poses and attributes independently. Furthermore, our model obtains better performance over existing methods for both pose estimation and attribute prediction tasks.
BibTeX
@article{park2018attribute,
  author = {Seyoung Park and Bruce Xiaohan Nie and Song-Chun Zhu},
  title = {{Attribute And-Or Grammar for Joint Parsing of Human Attributes, Part and Pose}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2018},
  url = {https://openreview.net/forum?id=siIC7Ep2r9},
}
Figure