Abstract
In this paper, we utilize structured learning to simultaneously address two intertwined problems: human pose estimation (HPE) and garment attribute classification (GAC), which are valuable for a variety of computer vision and multimedia applications. Unlike previous works that usually handle the two problems separately, our approach aims to produce a jointly optimal estimation for both HPE and GAC via a unified inference procedure. To this end, we adopt a preprocessing step to detect potential human parts from each image (i.e., a set of "candidates") that allows us to have a manageable input space. In this way, the simultaneous inference of HPE and GAC is converted to a structured learning problem, where the inputs are the collections of candidate ensembles, the outputs are the joint labels of human parts and garment attributes, and the joint feature representation involves various cues such as pose-specific features, garment-specific features, and cross-task features that encode correlations between human parts and garment attributes. Furthermore, we explore the "strong edge" evidence around the potential human parts so as to derive more powerful representations for oriented human parts. Such evidences can be seamlessly integrated into our structured learning model as a kind of energy function, and the learning process could be performed by standard structured Support Vector Machines (SVM) algorithm. However, the joint structure of the two problems is a cyclic graph, which hinders efficient inference. To resolve this issue, we compute instead approximate optima by using an iterative procedure, where in each iteration the variables of one problem are fixed. In this way, satisfactory solutions can be efficiently computed by dynamic programming. Experimental results on two benchmark datasets show the state-of-the-art performance of our approach.
Introduction
Human pose estimation (HPE) and garment attribute classification (GAC) are typically treated as independent tasks in computer vision pipelines. However, these two problems are fundamentally intertwined. For instance, the presence of a sleeveless shirt implies that skin-toned pixels should be expected in the lower arm regions, while the orientation of limbs provides critical spatial context for identifying garment features like necklines or sleeve lengths.
This paper introduces a unified structured learning framework that models these cross-task correlations. By solving both problems simultaneously, the proposed approach achieves higher accuracy than independent models, leveraging clothing as a physical constraint on pose detection and vice versa.
Methodology: Unified Structured Learning
The core of the approach is a Structured Support Vector Machine (SVM) that performs joint inference. The process begins with a preprocessing step where an existing pose estimator generates a set of "candidates" for each body part, significantly narrowing the search space.
The model utilizes three distinct feature types:
- Pose-specific features: Geometric constraints and Histograms of Oriented Gradients (HOG) descriptors.
- Garment-specific features: Modeling co-occurrence patterns of clothing attributes (e.g., certain necklines appearing with specific patterns).
- Cross-task features: Mappings that associate specific body regions (e.g., head and torso) with relevant garment attributes (e.g., neckline).
To improve limb detection, the authors incorporate "strong edge" evidence, rewarding pose candidates that align with parallel boundaries typical of human limbs. Because the joint dependency creates a complex cyclic graph, the authors employ an iterative inference procedure. By alternating between fixing garment attributes to solve for pose and fixing pose to solve for attributes, they achieve efficient solutions via dynamic programming.
Experimental Results
The framework was evaluated on the Buffy Dataset (748 video frames) and the DL (Daily Life) Dataset (1,000 web images).
Key performance metrics include:
- Pose Accuracy: The unified approach achieved a 91.4% Probability of Correct Part (PCP) on the Buffy dataset, surpassing the 89.1% baseline.
- Garment Classification: Significant improvements were noted in GAC; for example, sleeve classification precision rose from 51% to 61.7% on the Buffy dataset.
- Efficiency: The iterative inference process is computationally efficient, requiring approximately 2 seconds per image on a standard machine.
The results highlight that clothing acts as a powerful constraint; for example, identifying a "sleeveless" attribute helps the model distinguish skin-toned arms from background clutter that might otherwise confuse a standard pose estimator.
Conclusion
The proposed unified framework demonstrates that joint modeling of pose and garments leads to superior performance in both tasks. While the system remains robust across diverse clothing styles, it can still face challenges under heavy occlusion or extreme pose variations. Future work involves extending this structured learning approach to applications such as automated clothing retrieval and personalized fashion recommendations.
Summary updated