Look into Person: Joint Body Parsing & Pose Estimation Network and A New Benchmark

Xiaodan Liang, Ke Gong, Xiaohui Shen, Liang Lin

TL;DR

This paper introduces Look into Person (LIP), a massive new benchmark of over 50,000 images that unifies human parsing (pixel-level segmentation) and pose estimation (joint localization). The authors also propose JPPNet, a joint network that leverages the structural constraints of pose to improve parsing accuracy and vice versa, achieving state-of-the-art results on multiple benchmarks.

What Problem Does This Paper Address?

In the realm of human-centric computer vision, two tasks are paramount: human parsing (identifying which pixels belong to a "left arm" vs. a "jumpsuit") and pose estimation (locating the coordinates of the "elbow" or "knee").

Historically, these tasks have been treated as separate silos. Parsing models often lack a sense of human anatomy, leading to "anatomically impossible" results—like a right arm being attached to a left shoulder. Conversely, pose estimation models often struggle with occlusions or unusual viewpoints because they lack the fine-grained contextual information provided by segmentation.

The primary hurdles have been:

  1. Data Scarcity: Existing datasets were either small, limited to upright fashion models, or only provided one type of annotation (parsing or pose).
  2. Structural Ignorance: Standard segmentation networks (like DeepLab or FCNs) don't inherently "know" how a human body is put together.
Fig. 1. This example shows that human body structural information is helpful for human parsing. (a) The original image. (b) The parsing results by attention-to-scale [8], where the left arm is incorrectly labeled as the right arm. (c) Our parsing results successfully incorporate the structure information to generate reasonable outputs.

Methods

The authors propose the Joint Human Parsing and Pose Estimation Network (JPPNet). Its architecture is built on three pillars:

  1. Shared Feature Extraction: A ResNet-101 backbone (using atrous convolutions to maintain resolution) extracts a common set of features for both tasks.
  2. Task-Specific Subnets:
    • The Parsing Subnet uses Atrous Spatial Pyramid Pooling (ASPP) to capture multi-scale context.
  3. The Pose Subnet uses a series of convolutional layers to generate joint heatmaps.
  4. Iterative Refinement: This is the "secret sauce." The network takes the initial parsing and pose predictions, maps them back into a high-dimensional feature space, and refines them. This allows the model to use the predicted joint locations to "clean up" the segmentation boundaries and vice versa.

For scenarios where pose annotations are unavailable, the authors introduce SS-JPPNet (Self-Supervised JPPNet). This version calculates "pseudo-joints" by finding the center of mass of predicted parsing segments (e.g., the center of the "head" segment). It then uses a structure-sensitive loss to ensure the segmentation remains anatomically consistent without needing manual pose labels.

Datasets

The paper's most significant contribution to the community is the Look into Person (LIP) dataset.

  • Scale: 50,462 images (roughly 3x larger than previous parsing datasets).
  • Annotations: 19 semantic part labels (face, hair, skirt, etc.) and 16 body joints.
  • Diversity: Images are cropped from MS-COCO, featuring real-world complexity: occlusions, back views, and diverse poses.

The authors also evaluate their models on:

  • PASCAL-Person-Part: A smaller, coarser parsing dataset.
  • ATR: A dataset focused on fashion/upright persons.
  • MPII Human Pose: The standard benchmark for articulated pose.
Annotation examples from the LIP dataset compared to existing ones. LIP offers higher appearance variability and unified annotations.

Key Insights and Ideas

  • Mutual Benefit: The "aha" moment is that parsing and pose are complementary. Pose provides the global "skeleton" that prevents segmentation errors, while parsing provides the local "flesh" context that helps locate joints in cluttered backgrounds.
  • The "Head" Cue: Empirical analysis showed that the presence of a head/face is the most critical cue for current models. When the head is missing or the person is seen from the back, performance drops significantly across all baseline methods.
  • Self-Supervision via Geometry: You don't always need pose labels to learn structure. By simply enforcing that segments (like arms and legs) must have a coherent spatial center, the model learns a primitive but effective internal representation of human anatomy.

Key Results

  • Parsing: On the LIP validation set, JPPNet achieved a 51.37% Mean IoU, a massive jump over the 44.80% achieved by the previous state-of-the-art (DeepLab ResNet-101).
  • Pose: JPPNet reached 82.7% PCKh (Percentage of Correct Keypoints) on the LIP test set, outperforming standalone pose models like CPM and Hourglass.
  • Generalization: Models trained on LIP showed superior performance when tested on other datasets like ATR and MPII, proving that the diversity of LIP helps prevent overfitting.

Limitations and Future Work

Despite the gains, the authors note that back-view images and heavy occlusions remain significant challenges. In back-view cases, models still frequently confuse left and right limbs.

Future work could involve extending this unified framework to multi-person scenarios (which the authors have already begun collecting data for in a subset of LIP) and exploring more complex structural priors beyond simple joint centers.

Summary updated

Abstract

Human parsing and pose estimation have recently received considerable interest due to their substantial application potentials. However, the existing datasets have limited numbers of images and annotations and lack a variety of human appearances and coverage of challenging cases in unconstrained environments. In this paper, we introduce a new benchmark named "Look into Person (LIP)" that provides a significant advancement in terms of scalability, diversity, and difficulty, which are crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels and 16 body joints, which are captured from a broad range of viewpoints, occlusions, and background complexities. Using these rich annotations, we perform detailed analyses of the leading human parsing and pose estimation approaches, thereby obtaining insights into the successes and failures of these methods. To further explore and take advantage of the semantic correlation of these two tasks, we propose a novel joint human parsing and pose estimation network to explore efficient context modeling, which can simultaneously predict parsing and pose with extremely high quality. Furthermore, we simplify the network to solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into the parsing results without resorting to extra supervision. The dataset, code and models are available at http://www.sysu-hcp.net/lip/.
BibTeX
@article{liang2019look,
  author = {Xiaodan Liang and Ke Gong and Xiaohui Shen and Liang Lin},
  title = {{Look into Person: Joint Body Parsing \& Pose Estimation Network and A New Benchmark}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2019},
  doi = {10.1109/tpami.2018.2820063},
}
Figure