ProGraph: Temporally-alignable Probability Guided Graph Topological Modeling for 3D Human Reconstruction

Hongsheng Wang, Zehui Feng, Tong Xiao, Genfan Yang, Shengyu Zhang, Fei Wu, Feng LinarXiv 2024 · DOI · arXiv

Introduction

Reconstructing a 3D human mesh from a single 2D video remains a fundamental challenge in computer vision. The primary obstacles are self-occlusion—where body parts disappear behind objects or other limbs—and motion blur, which erases the fine details necessary for accurate pose estimation.

Existing approaches typically fall into two categories: regression-based methods, which attempt to map pixels directly to 3D coordinates, and rotation-based methods, which model the skeleton's joints. However, regression-based models often struggle with temporal consistency, while joint-only models are too sparse to capture the complex surface of the human body, frequently resulting in distorted meshes or "muscle stretching." ProGraph addresses these gaps by introducing a model that understands the underlying structural map (topology) of the human body to predict missing features across video sequences.

The ProGraph Framework

ProGraph introduces three interconnected modules designed to maintain physical consistency even under adverse conditions.

Graph Topological Modeling (GTM)

Rather than treating the body as a collection of independent points, GTM learns explicit relationships between vertices. By downsampling the standard 6,890-vertex SMPL human mesh to 431 key vertices and employing Graph Convolutions, the model maps structural connections. This ensures that if a limb is obscured by blur, the model can infer its position based on its known connection to the rest of the skeletal structure.

Temporally-alignable Probability Distribution (TPDist)

At the core of ProGraph is TPDist, which shifts from deterministic prediction to probabilistic modeling. Using a latent diffusion model, TPDist learns a probability distribution of potential body part locations across the entire video. A Temporal Transformer then aligns these probabilities across frames, ensuring that the resulting motion is smooth and physically plausible.

Overview of the ProGraph architecture, featuring Graph Topological Modeling (GTM) and Temporally-alignable Probability Distribution (TPDist).

Hierarchical Human Loss (HHLoss)

To refine the training process, the authors developed HHLoss. This function evaluates reconstruction at multiple levels of detail, assigning different weights to body parts based on their structural influence. This prevents localized errors in a limb from causing global distortions in the torso or overall posture.

Experimental Evaluation

The framework was evaluated on two primary benchmarks: Human3.6M, an indoor dataset with precise 3D ground truth, and 3DPW (3D Pose in the Wild), which features challenging outdoor scenes with natural occlusions and complex backgrounds. ProGraph was trained on a combination of both to ensure real-world robustness.

Key Results and Insights

ProGraph achieved state-of-the-art (SOTA) performance, particularly on the 3DPW dataset. Quantitatively, it reduced the Mean Per Vertex Error (MPVPE) by 3.1mm compared to the previous leading model, 4DHumans. On Human3.6M, it improved the Mean Per Joint Position Error (MPJPE) by 2.8mm over the baseline.

Qualitatively, ProGraph demonstrates significantly higher stability than competitors like PyMAF or GLoT. While other models often "lose" feet or distort the head during fast motion, ProGraph maintains a realistic human shape by effectively "hallucinating" missing data using its topological and probabilistic priors.

Qualitative comparison of ProGraph against state-of-the-art methods, demonstrating superior stability under occlusion and motion.

Conclusion

ProGraph demonstrates that explicit topological guidance and probabilistic diffusion are powerful tools for 3D human reconstruction. By modeling the probability of a feature's location rather than a fixed point, the system handles visual ambiguity with unprecedented grace. While extreme, prolonged occlusions remain a challenge, the framework sets a new standard for temporal consistency in monocular video reconstruction. Future work may focus on optimizing the computational efficiency of the diffusion process and extending the model to multi-person scenarios.

Summary updated

BibTeX
@misc{wang2024prograph,
  author = {Hongsheng Wang and Zehui Feng and Tong Xiao and Genfan Yang and Shengyu Zhang and Fei Wu and Feng Lin},
  title = {{ProGraph: Temporally-alignable Probability Guided Graph Topological Modeling for 3D Human Reconstruction}},
  year = {2024},
  eprint = {2411.04399},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.04399},
}
Figure