Feature Boosting Network For 3D Pose Estimation

Jun Liu, Henghui Ding, Amir Shahroudy, Ling-Yu Duan, Xudong Jiang, Gang Wang, Alex C. Kot

Abstract

Estimating 3D human pose from a single RGB image is a challenging task due to depth ambiguity and frequent self-occlusions. This paper introduces a Feature Boosting Network designed to exploit spatial dependencies between body joints. By integrating a Graphical ConvLSTM within a stacked Hourglass architecture, the model propagates information across a skeletal graph. Furthermore, a Context Consistency Gate (CCG) is employed to filter out unreliable local features caused by noise or occlusion. Experimental results on major benchmarks demonstrate that this approach significantly improves 3D pose estimation accuracy for both hands and full bodies.

Introduction

3D pose estimation from a single 2D image is a classic "ill-posed" problem in computer vision. Without depth information, networks must infer the third dimension from visual context and anatomical constraints. Standard convolutional neural networks (CNNs) often struggle when body parts are occluded or when articulations are complex.

In "Feature Boosting Network For 3D Pose Estimation," Liu et al. propose that anatomical relationships should be integrated directly into the feature-learning process rather than applied as a post-processing step. Their architecture "boosts" intermediate features by modeling the complex graphical relationships between joints, such as the physical connection between a wrist and an elbow or the symmetrical correlation between left and right knees.

Methodology

The core of the framework is the Long Short-Term Dependence-aware (LSTD) module, which is embedded within a stacked Hourglass CNN architecture. This module consists of two primary components: the Graphical ConvLSTM and the Context Consistency Gate (CCG).

The Feature Boosting Network architecture integrates LSTD and CCG modules within a stacked Hourglass CNN to refine joint features.

Graphical ConvLSTM

Unlike standard LSTMs that process data in a linear sequence, the Graphical ConvLSTM links units according to the actual topology of the human skeleton. This allows the network to propagate information across the body's graph, enabling it to estimate the position of occluded joints based on the known positions of neighboring joints. The graph accounts for both physical connections and symmetrical relations.

Context Consistency Gate (CCG)

To handle noisy or occluded features, the CCG acts as a trust mechanism. It predicts expected joint features based on global context and compares them to local convolutional evidence. If the local features are unreliable (e.g., a hand hidden behind the torso), the gate suppresses them and relies more on the context-based prediction to "boost" the representation.

Experiments and Results

The authors evaluated the model on several rigorous benchmarks, including the 3DHandPose dataset and Human3.6M (the gold standard for 3D body pose).

On the 3DHandPose dataset, the model achieved a Percentage of Correct Keypoints (PCK) of 89.5% at a 20mm threshold, a significant improvement over the baseline. On Human3.6M, the model achieved a mean Euclidean distance error of 61.10 mm, outperforming several competitive state-of-the-art models.

PCK curves on the 3DHandPose dataset demonstrate that the proposed 3D Pose Net with FB+ outperforms existing state-of-the-art methods.

Key Insights

  • Symmetry Matters: Modeling symmetrical connections (e.g., left hand to right hand) significantly improves accuracy, as their motions are often highly correlated.
  • Feature-Level Boosting: Refining intermediate features rather than output coordinates allows both 2D heatmap generation and 3D depth regression to benefit from cleaner data.
  • Context as a Filter: The CCG effectively turns the network into a self-correcting system by comparing local evidence against global context.

Conclusion

The Feature Boosting Network demonstrates that explicitly modeling graphical dependencies and using context to gate unreliable features can significantly advance 3D pose estimation. While hardware memory limits currently restrict the number of stacked sub-networks, the approach provides a robust foundation for future work in "in-the-wild" pose estimation and dynamic action-based skeletal graphs.

Summary updated

Abstract

In this paper, a feature boosting network is proposed for estimating 3D hand pose and 3D body pose from a single RGB image. In this method, the features learned by the convolutional layers are boosted with a new long short-term dependence-aware (LSTD) module, which enables the intermediate convolutional feature maps to perceive the graphical long short-term dependency among different hand (or body) parts using the designed Graphical ConvLSTM. Learning a set of features that are reliable and discriminatively representative of the pose of a hand (or body) part is difficult due to the ambiguities, texture and illumination variation, and self-occlusion in the real application of 3D pose estimation. To improve the reliability of the features for representing each body part and enhance the LSTD module, we further introduce a context consistency gate (CCG) in this paper, with which the convolutional feature maps are modulated according to their consistency with the context representations. We evaluate the proposed method on challenging benchmark datasets for 3D hand pose estimation and 3D full body pose estimation. Experimental results show the effectiveness of our method that achieves state-of-the-art performance on both of the tasks.
BibTeX
@article{liu2020feature,
  author = {Jun Liu and Henghui Ding and Amir Shahroudy and Ling-Yu Duan and Xudong Jiang and Gang Wang and Alex C. Kot},
  title = {{Feature Boosting Network For 3D Pose Estimation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2020},
  doi = {10.1109/tpami.2019.2894422},
}
Figure