Combining Parsing Information With Joint Structure for Human Pose Estimation

Shenming Feng, Xiying Li, Haifeng HuIEEE Access 2020 · OpenReview

Abstract

Recently, with the growing popularity of Convolutional Neural Networks (CNN), major progress has been made in human pose estimation. However, due to the lack of consideration of human body structure, current methods still suffer from occlusion, complex backgrounds, and large pose variations. Human parsing, as a highly relevant task, can provide useful semantic information of body parts for pose estimation. In this paper, we propose a novel convolutional network consisting of a pose encoder and a parsing encoder in parallel to combine parsing information with body structure information. The pose encoder extracts discriminative features containing rich structure information via a multi-branch parallel module and a hierarchical connection architecture. Simultaneously, the parsing encoder obtains valuable parsing information that serves as extra constraints. We also apply a joint classification loss for preserving structural consistency between keypoints and body parts. Experimental results on the extended PASCAL-Person-Part and LSP datasets demonstrate that performance can be greatly improved by utilizing parsing information.

What Problem Does This Paper Address?

Despite the success of CNNs, human pose estimation (HPE) still struggles with "hard" cases: severe occlusion (e.g., an arm behind a torso), cluttered backgrounds, and extreme body contortions.

The authors argue that many current models treat joints as isolated points, ignoring the fundamental biological constraint that joints belong to specific body parts. For instance, in a crowded scene, a model might mistake a nearby hip for a wrist. By ignoring the semantic "map" of the body (human parsing), these models lose a vital source of structural guidance that could correct these misalignments.

Motivation for using human parsing to correct pose estimation errors.

Methods

The proposed architecture consists of two parallel streams that eventually merge to produce final heatmap predictions:

1. The Pose Encoder

Based on the popular "Stacked Hourglass" design, this encoder includes two key upgrades:

  • Multi-Branch Parallel Module (MBPM): Instead of standard residual blocks, it uses multiple branches with different convolution scales and dilated convolutions to capture both local detail and global context simultaneously.
  • Hierarchical Connection Structure (HCS): This passes low-level structural features directly between different hourglass stages, preventing the loss of detail that usually occurs during repeated down-sampling and up-sampling.
The Multi-Branch Parallel Module (MBPM) used in the Pose Encoder.

2. The Parsing Encoder

This stream uses a Fully Convolutional Network (FCN) to segment the image into body parts (head, torso, arms, etc.). It employs a "shallow-to-deep" skip connection to ensure that fine-grained local features from early layers are preserved alongside high-level semantic features.

3. Joint Classification Loss ($L_{jc}$)

This novel loss function calculates the distance between a predicted joint and the center of its corresponding parsed body part. This forces the network to maintain structural consistency—essentially teaching the model that a "wrist" keypoint must be located within the "lower arm" segment.

Key Results and Insights

The model was evaluated on the Extended PASCAL-Person-Part and Extended Leeds Sports Pose (LSP) datasets. Key findings include:

  • Accuracy: The method reached 42.4 mAP on PASCAL-Person-Part, outperforming previous top-down and bottom-up methods.
  • Hard Joint Improvement: The model saw significant gains on difficult keypoints; specifically, a 7.0% boost for wrists and 3.3% for elbows compared to previous best-in-class methods.
  • Efficiency: Despite the dual-encoder setup, the top-down inference speed was roughly 3x faster than some competing architectures (2.5s vs 8s per image in certain configurations).
  • Complementary Tasks: Integrating pose estimation (where a joint is) with parsing (what that area is) creates a feedback loop that reduces anatomically impossible errors.

Limitations and Future Work

The primary weakness identified is scale sensitivity. When the human subject is very small relative to the total image size, the network struggles to learn the joint structure effectively. To address this, the authors suggest exploring super-resolution techniques to "zoom in" on small instances and investigating a more bidirectional approach where joint information further refines parsing masks.

Summary updated

Abstract

Recently, with the growing popularity of Convolutional Neural Networks (CNN), major progress has been made in human pose estimation. However, due to the lack of consideration of human body structure, the current methods still suffer from occlusion, complex background, and large pose variations. Human parsing, as a highly relevant task, can provide useful semantic information of body parts for pose estimation. In this paper, we propose a novel convolutional network, which consists of a pose encoder and a parsing encoder in parallel, to combine parsing information with body structure information for effectively assisting human pose estimation in unconstrained environment. The pose encoder can extract discriminative features that contain rich structure information by incorporating a multi-branch parallel module and a hierarchical connection architecture into hourglass network. On the other hand, the parsing encoder can obtain valuable parsing information that serves as extra constraints to help improve the performance of pose estimation. Then, the generated pose features and parsing features will be integrated together for heatmap prediction. In addition, we apply a joint classification loss for preserving the structural consistency between keypoints and body parts. By correctly localizing the joints in their corresponding body parts, the network can better optimize the spatial relationship between different keypoints, thus reducing the errors of estimation. It is worth mentioning that the proposed modules can be introduced into most of the existing methods. The experimental results on the extended PASCAL-Person-Part and LSP datasets demonstrate the effectiveness of our model, and the performance can be greatly improved by utilizing parsing information.
BibTeX
@inproceedings{feng2020combining,
  author = {Shenming Feng and Xiying Li and Haifeng Hu},
  title = {{Combining Parsing Information With Joint Structure for Human Pose Estimation}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=Maujey183p},
}
Figure