Joint Path Alignment Framework for 3D Human Pose and Shape Estimation From Video

Ji Woo Hong, Sunjae Yoon, Junyeong Kim, Chang D. YooIEEE Access 2023 · DOI · OpenReview

Introduction

Estimating 3D human pose and shape (3D-HPSE) from monocular video is a fundamental task for applications in virtual reality, sports analysis, and human-computer interaction. While single-frame estimators have achieved high accuracy, their application to video often results in jittery, unstable motion. Current video-based models typically address this by using temporal encoders and supervising "acceleration error" to penalize sudden movements. However, these methods often overlook "geometric misalignment," where a predicted motion path is consistently shifted or scaled incorrectly relative to the ground truth.

The Joint Path Alignment (JPA) Framework

The authors introduce the Joint Path Alignment (JPA) framework, a model-agnostic approach designed to correct systemic geometric biases in video-based 3D-HPSE. JPA can be integrated with various temporal encoders, such as GRUs or Transformers, to refine sequential predictions.

The core innovation is the Temporal Procrustes Alignment Regularization (TPAR) loss. While standard Procrustes Analysis (PA) is typically used to align static 3D poses, JPA applies it to sequences of joint positions, or "paths." This sequence-level supervision forces the model to learn the correct geometric shape and dynamics of the movement path, rather than just optimizing individual frames.

Predicted joint path without temporal consistency, illustrating the jitter and misalignment common in baseline models.

Methodology and Key Insights

The JPA framework operates through a sequence of steps designed to capture long-term geometric trends in motion:

  1. Sequence Prediction: The model generates a sequence of 3D joints over a temporal window.
  2. Path Alignment: TPAR loss performs Procrustes Analysis between the predicted path and the ground-truth path for each joint.
  3. Regularization: The difference between the original prediction and its Procrustes-aligned version is used as a loss term to train the temporal encoder.

A key insight of this work is that moving from per-frame or frame-to-frame supervision to sequence-level (path) supervision allows the model to identify and correct persistent offsets that simple acceleration-based smoothing might ignore. Because JPA acts as a loss-based regularizer during training, it can enhance any temporal model without requiring changes to its internal architecture.

Experimental Evaluation

The JPA framework was evaluated on three standard benchmarks:

  • 3DPW (3D Poses in the Wild): A challenging dataset featuring outdoor scenes with natural human motion captured by hand-held cameras.
  • MPI-INF-3DHP: An indoor dataset with multi-view markerless motion capture, including diverse activities.
  • Human3.6M: A widely used indoor dataset providing high-accuracy 3D ground truth from professional motion capture systems.

Quantitative Results

JPA consistently outperformed state-of-the-art methods across all datasets. On the 3DPW dataset, JPA improved the PA-MPJPE (Procrustes-aligned Mean Per Joint Position Error) from 52.1mm to 51.9mm while significantly reducing acceleration error. This demonstrates that JPA successfully breaks the traditional trade-off between temporal smoothness and per-frame accuracy.

Qualitative Results

Visual comparisons highlight the framework's ability to mitigate misalignments, particularly for extremities like wrists, which are prone to large errors in baseline models. JPA ensures that the smoothed path remains geometrically accurate, providing a more realistic representation of human motion.

Predicted joint path with temporal consistency, showing the improved alignment and smoothness achieved by the JPA framework.

Conclusion

The Joint Path Alignment framework provides a robust solution for enhancing the temporal consistency and geometric accuracy of 3D human pose estimation from video. By shifting from frame-level to sequence-level supervision via the TPAR loss, JPA effectively addresses systemic misalignments, setting a new standard for performance on major benchmarks. While sequence-level supervision increases training complexity, the resulting improvements in motion quality represent a significant advancement for video-based human motion analysis.

Summary updated

Abstract

3D human pose and shape estimation (3D-HPSE) from video aims to generate sequence of 3D mesh that depict human body in the video. Current deep learning based 3D-HPSE networks that takes video input have focused on improving temporal consistency among sequence of 3D joints by supervising acceleration error between predicted and ground-truth human motion. However, these methods overlooked the geometric misalignments of persistent discrepancy between geometric paths drawn by sequence of predicted joints and that of ground-truth joints. To this end, we propose Joint Path Alignment (JPA) framework, a model-agnostic approach that mitigates geometric misalignments by introducing Temporal Procrustes Alignment Regularization (TPAR) loss that performs group-wise sequence learning of joint movement paths. Unlike previous methods that rely solely on per-frame supervision for accuracy, our framework adds sequence-level accuracy supervision with TPAR loss by performing Procrustes analysis on the geometric paths drawn by sequences of predicted joints. Our experiments show that JPA framework advances the network to exceed the previous state-of-the-art performances on benchmark datasets in both per-frame accuracy and video smoothness metric.
BibTeX
@article{hong2023joint,
  author = {Ji Woo Hong and Sunjae Yoon and Junyeong Kim and Chang D. Yoo},
  title = {{Joint Path Alignment Framework for 3D Human Pose and Shape Estimation From Video}},
  journal = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3271285},
}
Figure