Not All Parts Are Created Equal: 3D Pose Estimation by Modeling Bi-Directional Dependencies of Body Parts

Jue Wang, Shaoli Huang, Xinchao Wang, Dacheng Tao

Abstract

Not all the human body parts have the same degree of freedom (DOF) due to the physiological structure. For example, the limbs may move more flexibly and freely than the torso does. Most of the existing 3D pose estimation methods, despite the very promising results achieved, treat the body joints equally and consequently often lead to larger reconstruction errors on the limbs. In this paper, we propose a progressive approach that explicitly accounts for the distinct DOFs among the body parts. We model parts with higher DOFs like the elbows, as dependent components of the corresponding parts with lower DOFs like the torso, of which the 3D locations can be more reliably estimated. Meanwhile, the high-DOF parts may, in turn, impose a constraint on where the low-DOF ones lie. As a result, parts with different DOFs supervise one another, yielding physically constrained and plausible pose-estimation results. To further facilitate the prediction of the high-DOF parts, we introduce a pose-attribution estimation, where the relative location of a limb joint with respect to the torso, which has the least DOF of a human body, is explicitly estimated and further fed to the joint-estimation module. The proposed approach achieves very promising results, outperforming the state of the art on several benchmarks.

Introduction

3D human pose estimation (HPE) from monocular images is a fundamental task in computer vision, yet it remains challenging due to depth ambiguity. A significant limitation of current state-of-the-art models is the uniform treatment of all body joints. In reality, human joints exhibit varying Degrees of Freedom (DOF). The torso, acting as a stable anchor, has significantly lower movement variance compared to distal joints like wrists and ankles. Ignoring these physiological constraints often results in high reconstruction errors and physically implausible poses for flexible limbs.

Overview of the progressive approach and pose attributes.

This research introduces a framework that respects the body's hierarchical structure. By modeling bi-directional dependencies between low-DOF and high-DOF parts, the system ensures that stable joints guide flexible ones, while flexible joints provide feedback constraints to maintain global skeletal integrity.

Methodology

The proposed framework consists of two primary stages: a multi-task feature extraction network and a bi-directional 3D pose estimation network.

Multi-Task Network and Pose Attributes

The first stage processes input images to predict 2D joint locations and a novel set of "pose attributes." These attributes classify the position of each limb joint relative to the torso plane as front, back, or on-plane. This approach transforms a complex 3D depth regression problem into a more manageable classification task, providing a robust geometric prior for the subsequent 3D refinement.

Bi-Directional 3D Pose Network

The 3D network is structured into two sequential blocks designed to model inter-part dependencies:

  1. Block-I (Progressive Dependency): This block estimates 3D coordinates in a hierarchical fashion, starting from the low-DOF torso, moving to proximal joints (elbows/knees), and finally to distal joints (wrists/ankles).
  2. Block-II (Constraint Feedback): To ensure physical plausibility, this block works in reverse, using the estimated positions of high-mobility limbs to refine the torso's position.
The bi-directional network architecture featuring multi-task learning and hierarchical blocks.

To handle "in-the-wild" images where 3D data is unavailable, the authors employ Unsupervised Domain Adaptation. A domain classifier is used alongside a gradient reversal layer to ensure that features extracted from laboratory datasets (with 3D ground truth) and unlabelled real-world images are indistinguishable.

Experimental Results

The model was evaluated on three major benchmarks: Human3.6M, MPII, and MPI-INF-3DHP.

  • Quantitative Performance: On the Human3.6M dataset, the method achieved a Mean Per Joint Position Error (MPJPE) of 52.6mm under Protocol #1, surpassing existing state-of-the-art methods such as Martinez et al. and Pavlakos et al.
  • Limb Accuracy: The inclusion of pose attributes led to the most significant error reductions in distal joints (wrists and ankles), which typically suffer from the highest uncertainty in standard regression models.
  • Generalization: Testing on the MPI-INF-3DHP dataset yielded a 71.9% 3DPCK, confirming the effectiveness of the domain adaptation strategy for outdoor and complex scenes.

Conclusion

By acknowledging that "not all parts are created equal," this work demonstrates that modeling physiological hierarchies and bi-directional dependencies significantly improves 3D pose accuracy. The introduction of pose attributes as a classification task effectively mitigates depth ambiguity, leading to more stable and physically sound human pose reconstructions.

Summary updated

Abstract

Not all the human body parts have the same degree of freedom (DOF) due to the physiological structure. For example, the limbs may move more flexibly and freely than the torso does. Most of the existing 3D pose estimation methods, despite the very promising results achieved, treat the body joints equally and consequently often lead to larger reconstruction errors on the limbs. In this paper, we propose a progressive approach that explicitly accounts for the distinct DOFs among the body parts. We model parts with higher DOFs like the elbows, as dependent components of the corresponding parts with lower DOFs like the torso, of which the 3D locations can be more reliably estimated. Meanwhile, the high-DOF parts may, in turn, impose a constraint on where the low-DOF ones lie. As a result, parts with different DOFs supervise one another, yielding physically constrained and plausible pose-estimation results. To further facilitate the prediction of the high-DOF parts, we introduce a pose-attribution estimation, where the relative location of a limb joint with respect to the torso, which has the least DOF of a human body, is explicitly estimated and further fed to the joint-estimation module. The proposed approach achieves very promising results, outperforming the state of the art on several benchmarks.
BibTeX
@inproceedings{wang2019not,
  author = {Jue Wang and Shaoli Huang and Xinchao Wang and Dacheng Tao},
  title = {{Not All Parts Are Created Equal: 3D Pose Estimation by Modeling Bi-Directional Dependencies of Body Parts}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=ggS2DcROY7},
}
Figure