Monocular vision 3D human posture estimation network based on multi-joint feature fusion

Di Jia, Dongrui He, Xuefeng Han, Yang Liu, Shuo Cheng, Yuqi LiuJournal of Image and Graphics 2025 · DOI

Abstract

Estimating 3D human poses from a single 2D camera view is a classic "ill-posed" problem due to depth ambiguity and self-occlusion. This paper proposes a novel deep learning architecture that addresses temporal redundancy and the loss of subtle joint movements through multi-joint feature fusion. By integrating a multi-branch temporal convolution module for motion extraction, a joint feature fusion module for inter-limb relationships, and a joint restraint module for spatial consistency, the network achieves significant accuracy gains. On the Human3.6M dataset, the method achieves a Mean Per Joint Position Error (MPJPE) of 29.0 mm, representing a 4.9% improvement over state-of-the-art baselines, with even higher performance in complex action sequences.

Introduction

Monocular 3D human pose estimation (HPE) is critical for applications in augmented reality, sports analysis, and human-computer interaction. However, current models often struggle with two main issues: temporal redundancy, where adjacent video frames provide overlapping information that can "wash out" details, and the failure to capture minute joint movements. This research introduces a three-pronged architecture designed to rethink how joint features are fused across time and space to produce more anatomically grounded 3D poses.

Methodology

The proposed network architecture is built around three core modules that work in tandem to refine the 3D estimation process:

  1. Multi-Branch Joint Motion Feature Extraction: Instead of a single path, the network uses four parallel branches with strided convolutions at different intervals. This allows the model to capture motion at multiple temporal scales, filtering out redundancy while highlighting subtle changes like wrist flicks or balance shifts.
  2. Joint Feature Fusion Module: This module models the high-dimensional relationships between different body parts. By grouping joints into five topological categories (torso, left/right arms, and left/right legs), the network learns how the movement of one limb relates to another, ensuring cohesive body motion.
  3. Joint Restraint Module: Acting as a "sanity check," this module uses the 2D joint positions from the middle frame as an implicit constraint. By fusing this 2D evidence with the initial 3D estimate, the network prunes unrealistic 3D poses that do not align with the observed 2D image.
Overall architecture of the proposed 3D human pose estimation network.

A key insight of this approach is that topology matters. By grouping joints based on the human skeleton's actual structure rather than treating them as independent points, the model learns more meaningful spatial relationships.

Joint relationship and grouping diagrams used for feature fusion.

Results and Discussion

The model was rigorously evaluated on three gold-standard datasets: Human3.6M, MPI-INF-3DHP, and HumanEva.

  • Human3.6M: The network achieved an MPJPE of 29.0 mm. The improvement was most pronounced in difficult scenarios; for "Sitting Down," the error dropped by 7.7%, and for "Walking Dog," it dropped by 8.2%.
  • MPI-INF-3DHP: The model showed exceptional generalization, with a 36.2% reduction in MPJPE and a 12.9% increase in the Area Under the Curve (AUC) metric.
  • Efficiency: Despite the multi-branch approach, the model maintains a competitive parameter count of approximately 30M, which is lower than several heavy-duty transformer-based models like MixSTE.

Conclusion

By effectively fusing multi-scale temporal features and enforcing 2D spatial constraints, the proposed network significantly improves the accuracy and robustness of monocular 3D human pose estimation. Future work will focus on model compression and reducing input sequence lengths to enable real-time applications in latency-sensitive environments.

Summary updated

Abstract

目的针对目前三维人体姿态估计方法未能有效处理时间序列冗余,难以捕获人体关节上微小变化的问题,提出一种融合多关节特征的单目视觉三维人体姿态估计网络。方法在关节运动特征提取模块中,采用多分支操作提取关节在时间维度上的运动特征,并将不同特征融合形成具有高度表达力的特征表示。关节特征融合模块整合了不同关节组和中间帧的全局信息,通过矩阵内积的方式表达不同关节组在高纬度空间的相对位置及相互联系,得到中间3D姿态的初估值。关节约束模块引入中间帧的2D关节点空间位置关系作为隐式约束,与中间帧3D姿态初估值融合,减少不合理的姿态输出,提高最终3D姿态估计的准确性。结果实验结果表明,与MHFormer方法相比,本文方法在Human3.6M数据集上的平均关节位置误差(mean per joint position error,MPJPE)结果为29.0 mm,误差降低4.9%,对于复杂动作,如SittingDown和WalkDog,误差降低了7.7%和8.2%。在MPI-INF-3DHP数据集上,MPJPE指标降低36.2%,曲线下面积(area under the curve,AUC)指标提升12.9%,正确关节点百分比(percentage of correct keypoints,PCK)指标提升3%。实验结果体现出在面对复杂动作问题时,网络利用各分支提取了不同的关节时序运动特征,将不同关节组的位置信息进行融合交互,结合当前帧的关节姿态信息加以约束,取得更高的精度。在HumanEva数据集上的实验结果表明了本文方法适用不同数据集,消融实验进一步验证了各个模块的有效性。结论本文网络有效地融合了人体多关节特征,可以更好地提高单目视觉三维人体姿态估计的准确性,且具备较高的泛化性。
BibTeX
@article{jia2025monocular,
  author = {Di Jia and Dongrui He and Xuefeng Han and Yang Liu and Shuo Cheng and Yuqi Liu},
  title = {{Monocular vision 3D human posture estimation network based on multi-joint feature fusion}},
  journal = {Journal of Image and Graphics},
  year = {2025},
  doi = {10.11834/jig.240672},
}
Figure