Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark.

Xiaodan Liang, Ke Gong, Xiaohui Shen, Liang Lin

TL;DR

This paper introduces "Look into Person" (LIP), a large-scale, diverse, and challenging benchmark dataset for human parsing and pose estimation. They also propose a novel joint network (JPPNet) that simultaneously predicts both tasks, leveraging their inherent correlations for improved accuracy, and a simplified self-supervised approach for human parsing.

What Problem Does This Paper Address?

Understanding human bodies in images is crucial for many applications like surveillance and behavior analysis. Two key tasks in this area are human parsing (segmenting an image into fine-grained body parts and clothing) and human pose estimation (locating key body joints). While significant progress has been made using deep learning, existing datasets suffer from limited scale, diversity, and annotations, especially for challenging scenarios like occlusions, varied viewpoints, and complex backgrounds. This hinders the development of robust models. Furthermore, these two tasks are highly correlated but often addressed separately, missing opportunities for mutual benefit.

Methods

The authors introduce two main methodological contributions:

  • Joint Human Parsing and Pose Estimation Network (JPPNet): This network leverages a shared deep residual network for feature extraction. It then uses two distinct sub-networks for parsing and pose estimation, followed by a refinement network. The key innovation is the explicit modeling of the semantic correlation between parsing and pose, allowing them to mutually inform each other. JPPNet incorporates two "coarse-to-fine" strategies: multi-scale feature processing and iterative refinement.
  • Self-supervised Structure-sensitive Learning (SS-JPPNet): For scenarios where pose annotations are unavailable (common in many existing datasets), the authors propose a simplified human parsing network. This approach introduces a novel "structure-sensitive loss" that uses the predicted parsing results to infer human joint structures. This inferred structure then acts as a self-supervision signal to improve the parsing accuracy without requiring explicit pose annotations.

Datasets

  • Look into Person (LIP) Dataset: This is the primary contribution. It's a new, large-scale benchmark featuring:
  • 50,462 annotated images.
  • 19 semantic part labels for human parsing.
  • 16 body joint locations for pose estimation.
  • High diversity in viewpoints, occlusions, appearances, and background complexity.
  • A public evaluation server to prevent overfitting.
  • Existing Datasets: The paper also uses and compares against results on:
  • PASCAL-Person-Part [9] (for human parsing).
  • MPII Human Pose dataset [1] (for human pose estimation).
  • ATR [26] and LSP [19] (mentioned as existing datasets for comparison).

Key Insights and Ideas

  • Human Parsing and Pose Estimation are Complementary: The paper strongly advocates for the joint modeling of these tasks, showing that incorporating structural information from pose significantly improves parsing, and vice-versa.
  • Dataset Scale and Diversity are Crucial: The LIP dataset highlights the limitations of previous datasets and demonstrates the need for more challenging benchmarks to push the state-of-the-art.
  • Structure-Sensitive Loss for Unsupervised Parsing: The self-supervised approach is a clever way to leverage inherent human structure even without explicit pose labels, making parsing models more robust and adaptable.
  • Head is a Critical Cue: Analysis shows that head-missing images and back views are particularly challenging for both parsing and pose estimation, emphasizing the importance of contextual information.

Key Results

  • LIP Dataset Performance: Existing state-of-the-art methods show significantly lower performance on LIP compared to other benchmarks, underscoring its difficulty. For instance, the best previous method achieved a mean IoU of 44.80% on LIP, while the proposed JPPNet reached 51.37%.
  • JPPNet Performance: The proposed JPPNet achieved state-of-the-art results on both human parsing (e.g., 51.37% mean IoU on LIP validation) and pose estimation (e.g., 82.7% PCKh on LIP test).
  • SS-JPPNet Performance: The self-supervised SS-JPPNet also significantly outperformed existing methods on the PASCAL-Person-Part dataset (achieving 59.36% IoU), demonstrating its effectiveness even without pose annotations.
  • Challenge Analysis: The paper provides detailed analyses showing how factors like occlusion, back view, and head-missing images impact performance, offering valuable insights into model weaknesses.

Limitations and Future Work

  • Head Dependency: The analysis reveals a strong reliance on head information for both tasks. Future work could focus on reducing this dependency.
  • Refinement Impact: While iterative refinement benefits pose estimation, it slightly degrades parsing performance in the proposed JPPNet. Further investigation into optimizing this trade-off is suggested.
  • Multi-person Scenarios: While the LIP dataset includes multi-person images, the primary focus of the proposed JPPNet is on single-person analysis. Extending the joint framework to handle multiple interacting people would be a valuable future direction.
  • Dataset Expansion: The authors suggest that the LIP dataset can be further expanded with more challenging scenarios and annotations.

Summary updated

Abstract

Human parsing and pose estimation have recently received considerable interest due to their substantial application potentials. However, the existing datasets have limited numbers of images and annotations and lack a variety of human appearances and coverage of challenging cases in unconstrained environments. In this paper, we introduce a new benchmark named “Look into Person (LIP)” that provides a significant advancement in terms of scalability, diversity, and difficulty, which are crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels and 16 body joints, which are captured from a broad range of viewpoints, occlusions, and background complexities. Using these rich annotations, we perform detailed analyses of the leading human parsing and pose estimation approaches, thereby obtaining insights into the successes and failures of these methods. To further explore and take advantage of the semantic correlation of these two tasks, we propose a novel joint human parsing and pose estimation network to explore efficient context modeling, which can simultaneously predict parsing and pose with extremely high quality. Furthermore, we simplify the network to solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into the parsing results without resorting to extra supervision. The datasets, code and models are available at http://www.sysu-hcp.net/lip/.
BibTeX
@article{liang2019look,
  author = {Xiaodan Liang and Ke Gong and Xiaohui Shen and Liang Lin},
  title = {{Look into Person: Joint Body Parsing \& Pose Estimation Network and a New Benchmark.}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2019},
  doi = {10.1109/tpami.2018.2820063},
}
Figure