PhaseMP: Robust 3D Pose Estimation via Phase-conditioned Human Motion Prior

Mingyi Shi, Sebastian Starke, Yuting Ye, Taku Komura, Jungdam Won

Introduction

Estimating 3D human motion from 2D video or sparse sensors—such as VR trackers—remains a fundamental challenge in computer vision. While deep learning has significantly advanced the field, existing models often struggle in "in-the-wild" scenarios. When a subject is occluded by furniture or moves out of the camera's frame, temporal models frequently lose track, resulting in jittery, physically impossible, or "teleporting" poses.

The core issue is a lack of robust temporal momentum. Most current models focus on short-term frame-to-frame transitions without capturing the underlying rhythmic "beat" of human movement. PhaseMP addresses this by integrating frequency-domain "phase" features to act as a form of temporal momentum, ensuring stable and physically plausible 3D pose estimation even during heavy occlusion.

PhaseMP handles various challenging scenarios including occlusion, sparse data, and noisy inputs through motion in-betweening.

Methodology: Phase-conditioned Motion Prior

PhaseMP (Phase-conditioned Motion Prior) introduces a novel architecture built on three primary pillars:

  1. Deep Phase Features: The system utilizes a pre-trained Periodic Autoencoder (PAE) to map a window of joint velocities into the frequency domain. By extracting amplitude, frequency, and phase shift, the model captures the rhythmic "momentum" of the motion.
  2. Phase-conditioned CVAE: Unlike previous state-of-the-art models that rely solely on the preceding pose, PhaseMP employs a Conditional Variational Autoencoder (CVAE) conditioned on the phase feature. This ensures that predicted transitions align with the current rhythmic momentum.
  3. Robust Inference with Dynamic Interpolation: During inference, the model employs a confidence-aware update rule. High-quality input data updates the phase based on observations, while missing or noisy data triggers a "cyclic update," allowing the internal rhythm to carry the motion forward until visual evidence is restored.
  4. SirenMLP Decoder: The decoder utilizes sinusoidal activation functions (SIREN), which are inherently better suited for representing periodic human motion than standard ReLU activations.

Experimental Evaluation

The model was trained on the AMASS database and evaluated using standard benchmarks like HumanEva, as well as real-world datasets i3DB and PROX, which feature complex human-environment interactions.

Key Results

PhaseMP demonstrates a significant lead over existing methods like HuMoR and VIBE, particularly in degraded data scenarios:

  • Accuracy: On the i3DB dataset, PhaseMP achieved a P-MPJPE of 14.19cm, outperforming HuMoR (14.51cm) and VIBE (15.08cm).
  • Physical Realism: The model significantly reduced "foot sliding," achieving a foot contact error (P-Frep) of 2.03%, compared to HuMoR's 2.12% and VIBE's 7.98%.
  • Qualitative Stability: In scenarios with heavy occlusion, such as a person walking behind a sofa, PhaseMP maintains a smooth, realistic gait while baseline models often collapse.
PhaseMP demonstrates robust 3D pose estimation from real-world video sequences involving complex interactions and occlusions.

Discussion and Future Work

PhaseMP's primary innovation is treating frequency-domain features as a proxy for physical momentum. Because phase is unidirectional, it naturally prevents the "backward" glitches common in other temporal models. Furthermore, frequency-domain features remain more stable over time than raw joint positions, providing a reliable anchor for optimization.

Limitations: The current system assumes a flat ground plane and a static camera. Additionally, phase prediction can eventually drift during excessively long periods of total invisibility (several seconds).

Future Directions: Potential improvements include using phase features as positional encodings in Transformer architectures or integrating multi-modal signals, such as the audio of footsteps, to further constrain the phase estimation.

Summary updated

Abstract

We present a novel motion prior, called PhaseMP, modeling a probability distribution on pose transitions conditioned by a frequency domain feature extracted from a periodic autoencoder. The phase feature further enforces the pose transitions to be unidirectional (i.e. no backward movement in time), from which more stable and natural motions can be generated. Specifically, our motion prior can be useful for accurately estimating 3D human motions in the presence of challenging input data, including long periods of spatial and temporal occlusion, as well as noisy sensor measurements. Through a comprehensive evaluation, we demonstrate the efficacy of our novel motion prior, showcasing its superiority over existing state-of-the-art methods by a significant margin across various applications, including video-to-motion and motion estimation from sparse sensor data, and etc.
BibTeX
@inproceedings{shi2023phasemp,
  author = {Mingyi Shi and Sebastian Starke and Yuting Ye and Taku Komura and Jungdam Won},
  title = {{PhaseMP: Robust 3D Pose Estimation via Phase-conditioned Human Motion Prior}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01353},
}
Figure