Double-chain Constraints for 3D Human Pose Estimation in Images and Videos

Hongbo Kang, Yong Wang, Mengyuan Liu, Doudou Wu, Peng Liu, Wenming YangarXiv 2023 · DOI · arXiv

Abstract

Estimating 3D human poses from 2D images or video sequences is a challenging "ill-posed" problem due to depth ambiguity. This paper presents DC-GCT (Double-chain Graph Convolutional Transformer), a novel architecture that simultaneously models local skeletal constraints and global body dependencies. By utilizing a parallel "double-chain" structure—comprising Local-to-Global and Global-to-Local pathways—the model achieves state-of-the-art accuracy on major benchmarks like Human3.6M and MPI-INF-3DHP. Notably, DC-GCT maintains high computational efficiency, making it suitable for real-time video applications.

Introduction

The process of "2D-to-3D lifting" involves reconstructing a 3D skeleton from 2D joint coordinates. Historically, researchers have relied on two primary methods: Graph Convolutional Networks (GCNs), which excel at modeling local physical constraints (e.g., limb proportions), and Transformers, which capture global dependencies (e.g., cross-body coordination). However, GCNs often fail to grasp long-range relationships, while Transformers can overlook the inherent structural logic of the human skeleton.

Conceptual overview of global and local constraints within the human skeleton.

Furthermore, many existing multi-frame models for video are computationally intensive, hindering their deployment in real-time scenarios such as Augmented Reality (AR) or human-computer interaction.

Methodology: The DC-GCT Architecture

The core innovation of this work is the Double-chain Constraint mechanism. Unlike sequential models, DC-GCT processes features through two parallel pipelines to capture a multi-level representation of the human body:

  1. Local-to-Global Chain: This path utilizes a Local Constraint Module (LCM) based on GCNs to model immediate joint neighbors, followed by a Global Constraint Module (GCM) to integrate broader body context.
  2. Global-to-Local Chain: This inverse path ensures that global positioning informs the refinement of local joint details.
The DC-GCT architecture, illustrating the parallel local and global constraint modules and the feature interaction module.

To synchronize these chains, the authors developed a Feature Interaction Module (FIM) that fuses and maps features between the two paths, preventing information loss during the transition between local and global scales. For video data, a Temporal Embedding method is introduced that "guides" the embedding of a sequence using joint data from the target frame, providing temporal context with minimal computational overhead.

Experimental Results

The model was rigorously evaluated on the Human3.6M and MPI-INF-3DHP datasets. Key findings include:

  • State-of-the-Art Accuracy: On Human3.6M, DC-GCT achieved a Mean Per Joint Position Error (MPJPE) of 48.4 mm using detected 2D poses, outperforming previous methods across all action categories. With ground truth 2D locations, the error dropped to 32.4 mm.
  • Efficiency: DC-GCT is significantly more lightweight than its predecessors. It utilizes approximately 2.3M parameters, compared to 9.6M for Poseformer and 18.9M for other leading models, while delivering superior performance.
  • Temporal Consistency: The model maintains high accuracy in video sequences with a negligible increase in FLOPs compared to single-frame models.

Conclusion and Future Work

DC-GCT demonstrates that parallelizing local and global constraint modeling is highly effective for 3D pose estimation. While the model sets new benchmarks, challenges remain in scenarios involving heavy occlusion or extreme poses that deviate significantly from training data. Future research will focus on enhancing the architecture's robustness to "in-the-wild" conditions and improving handling for partial body detections.

Summary updated

Abstract

Reconstructing 3D poses from 2D poses lacking depth information is particularly challenging due to the complexity and diversity of human motion. The key is to effectively model the spatial constraints between joints to leverage their inherent dependencies. Thus, we propose a novel model, called Double-chain Graph Convolutional Transformer (DC-GCT), to constrain the pose through a double-chain design consisting of local-to-global and global-to-local chains to obtain a complex representation more suitable for the current human pose. Specifically, we combine the advantages of GCN and Transformer and design a Local Constraint Module (LCM) based on GCN and a Global Constraint Module (GCM) based on self-attention mechanism as well as a Feature Interaction Module (FIM). The proposed method fully captures the multi-level dependencies between human body joints to optimize the modeling capability of the model. Moreover, we propose a method to use temporal information into the single-frame model by guiding the video sequence embedding through the joint embedding of the target frame, with negligible increase in computational cost. Experimental results demonstrate that DC-GCT achieves state-of-the-art performance on two challenging datasets (Human3.6M and MPI-INF-3DHP). Notably, our model achieves state-of-the-art performance on all action categories in the Human3.6M dataset using detected 2D poses from CPN, and our code is available at: https://github.com/KHB1698/DC-GCT.
BibTeX
@misc{kang2023double,
  author = {Hongbo Kang and Yong Wang and Mengyuan Liu and Doudou Wu and Peng Liu and Wenming Yang},
  title = {{Double-chain Constraints for 3D Human Pose Estimation in Images and Videos}},
  year = {2023},
  eprint = {2308.05298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.05298},
}
Figure