Unified Structured Learning for Simultaneous Human Pose Estimation and Garment Attribute Classification

Jie Shen, Guangcan Liu, Jia Chen, Yuqiang Fang, Jianbin Xie, Yong Yu, Shuicheng Yan

TL;DR

This paper introduces a unified structured learning framework that simultaneously tackles human pose estimation (HPE) and garment attribute classification (GAC). By modeling the intertwined relationship between how a person is standing and what they are wearing, the authors demonstrate that solving both problems together leads to higher accuracy than treating them as independent tasks.

What Problem Does This Paper Address?

In computer vision, human pose estimation (identifying the location and orientation of body parts) and garment attribute classification (identifying clothing features like sleeve length or neckline type) are usually treated as separate pipelines.

However, these two tasks are naturally "intertwined." For example, if a system knows a person is wearing a sleeveless shirt, it shouldn't expect to find fabric-colored pixels where the lower arms are. Conversely, knowing the exact orientation of the arms helps the system focus on the correct image regions to identify sleeve attributes. The authors address the challenge of creating a mathematical model that captures these cross-task correlations while remaining computationally efficient.

Methods

The authors propose a Unified Structured Learning approach based on a Structured Support Vector Machine (SVM).

  1. Preprocessing: Instead of looking at every pixel, they use an existing pose estimator to generate a set of "candidates" for each body part. This narrows the search space significantly.
  2. Feature Design: They use three types of features:
  3. Pose-specific: Standard descriptors like Histograms of Oriented Gradients (HOG) and geometric constraints between connected parts (e.g., an elbow must be near a shoulder).
  4. Garment-specific: Modeling which attributes often appear together (e.g., a "round" neckline often co-occurs with "T-shirt" patterns).
  5. Cross-task: The core innovation—features that encode the correlation between specific body parts and specific garment attributes (e.g., associating the torso and head regions with the "neckline" attribute).
  6. Strong Edge Evidence: They incorporate a "strong edge" energy function. Since human limbs are roughly cylindrical, they often produce parallel edges in an image. The model rewards pose candidates that align with these detected boundaries.
  7. Iterative Inference: The joint dependency between pose and clothing creates a "cyclic graph," which is notoriously hard to solve. To fix this, the authors use an iterative procedure: they fix the garment attributes and solve for the best pose, then fix the pose and solve for the best garment attributes. This allows them to use efficient dynamic programming at each step.

Datasets

The model was evaluated on two primary benchmarks:

  • Buffy Dataset: 748 annotated video frames from the TV show Buffy the Vampire Slayer. While originally a pose dataset, the authors manually added garment attribute labels.
  • DL (Daily Life) Dataset: A new dataset introduced by the authors containing 1,000 photos from the web. It features more diverse clothing styles and real-world variations than the Buffy dataset.

Key Insights and Ideas

  • The "Aha" Moment on Constraints: The most interesting insight is that clothing acts as a physical constraint on pose detection. In the paper's examples, a "sleeveless" attribute helps the model correctly identify skin-toned lower arms, whereas a standard pose estimator might get confused by background clutter.
  • Global vs. Local Learning: Most previous works learn pose and clothing locally (separately). This paper argues for "learning globally and inferring locally," allowing the model to understand the "big picture" of the human appearance.
  • Edge Refinement: Simply looking at HOG features isn't enough for precise limb orientation. By explicitly looking for "strong edges" (parallel boundaries), the model significantly reduces errors in detecting lower arms, which are often the most difficult parts to track.

Key Results

  • Pose Accuracy: On the Buffy dataset, the unified approach achieved a 91.4% Probability of Correct Part (PCP), outperforming the state-of-the-art "YDR" baseline (89.1%).
  • Garment Classification: The joint model showed a significant boost in GAC. For example, on the Buffy dataset, the precision for "sleeve" classification jumped from roughly 51% (using baseline methods) to 61.7%.
  • Efficiency: Despite the complexity of the joint model, the iterative inference is fast, taking only about 2 seconds per image on a standard machine.

Limitations and Future Work

The authors acknowledge that the system can still be "confused" in two main scenarios:

  1. Heavy Occlusion: When body parts are hidden behind objects or other people, the geometric and garment constraints may not be enough to recover the pose.
  2. Extreme Variation: Very unusual poses or highly unconventional fashion can fall outside the correlations learned by the model.

For future work, the authors suggest extending this framework to more complex applications like automated online clothing retrieval and personalized fashion recommendation systems.

Summary updated

Abstract

In this paper, we utilize structured learning to simultaneously address two intertwined problems: 1) human pose estimation (HPE) and 2) garment attribute classification (GAC), which are valuable for a variety of computer vision and multimedia applications. Unlike previous works that usually handle the two problems separately, our approach aims to produce an optimal joint estimation for both HPE and GAC via a unified inference procedure. To this end, we adopt a preprocessing step to detect potential human parts from each image (i.e., a set of candidates) that allows us to have a manageable input space. In this way, the simultaneous inference of HPE and GAC is converted to a structured learning problem, where the inputs are the collections of candidate ensembles, outputs are the joint labels of human parts and garment attributes, and joint feature representation involves various cues such as pose-specific features, garment-specific features, and cross-task features that encode correlations between human parts and garment attributes. Furthermore, we explore the strong edge evidence around the potential human parts so as to derive more powerful representations for oriented human parts. Such evidences can be seamlessly integrated into our structured learning model as a kind of energy function, and the learning process could be performed by standard structured support vector machines algorithm. However, the joint structure of the two problems is a cyclic graph, which hinders efficient inference. To resolve this issue, we compute instead approximate optima using an iterative procedure, where in each iteration, the variables of one problem are fixed. In this way, satisfactory solutions can be efficiently computed by dynamic programming. Experimental results on two benchmark data sets show the state-of-the-art performance of our approach.
BibTeX
@article{shen2014unified,
  author = {Jie Shen and Guangcan Liu and Jia Chen and Yuqiang Fang and Jianbin Xie and Yong Yu and Shuicheng YAN},
  title = {{Unified Structured Learning for Simultaneous Human Pose Estimation and Garment Attribute Classification}},
  journal = {IEEE Trans. Image Process.},
  year = {2014},
  url = {https://openreview.net/forum?id=CDGIkCCJo-n},
}

Loading related content...

Figure