Introduction
Estimating 3D human motion from 2D video or sparse sensors—such as VR trackers—remains a fundamental challenge in computer vision. While deep learning has significantly advanced the field, existing models often struggle in "in-the-wild" scenarios. When a subject is occluded by furniture or moves out of the camera's frame, temporal models frequently lose track, resulting in jittery, physically impossible, or "teleporting" poses.
The core issue is a lack of robust temporal momentum. Most current models focus on short-term frame-to-frame transitions without capturing the underlying rhythmic "beat" of human movement. PhaseMP addresses this by integrating frequency-domain "phase" features to act as a form of temporal momentum, ensuring stable and physically plausible 3D pose estimation even during heavy occlusion.
Methodology: Phase-conditioned Motion Prior
PhaseMP (Phase-conditioned Motion Prior) introduces a novel architecture built on three primary pillars:
- Deep Phase Features: The system utilizes a pre-trained Periodic Autoencoder (PAE) to map a window of joint velocities into the frequency domain. By extracting amplitude, frequency, and phase shift, the model captures the rhythmic "momentum" of the motion.
- Phase-conditioned CVAE: Unlike previous state-of-the-art models that rely solely on the preceding pose, PhaseMP employs a Conditional Variational Autoencoder (CVAE) conditioned on the phase feature. This ensures that predicted transitions align with the current rhythmic momentum.
- Robust Inference with Dynamic Interpolation: During inference, the model employs a confidence-aware update rule. High-quality input data updates the phase based on observations, while missing or noisy data triggers a "cyclic update," allowing the internal rhythm to carry the motion forward until visual evidence is restored.
- SirenMLP Decoder: The decoder utilizes sinusoidal activation functions (SIREN), which are inherently better suited for representing periodic human motion than standard ReLU activations.
Experimental Evaluation
The model was trained on the AMASS database and evaluated using standard benchmarks like HumanEva, as well as real-world datasets i3DB and PROX, which feature complex human-environment interactions.
Key Results
PhaseMP demonstrates a significant lead over existing methods like HuMoR and VIBE, particularly in degraded data scenarios:
- Accuracy: On the i3DB dataset, PhaseMP achieved a P-MPJPE of 14.19cm, outperforming HuMoR (14.51cm) and VIBE (15.08cm).
- Physical Realism: The model significantly reduced "foot sliding," achieving a foot contact error (P-Frep) of 2.03%, compared to HuMoR's 2.12% and VIBE's 7.98%.
- Qualitative Stability: In scenarios with heavy occlusion, such as a person walking behind a sofa, PhaseMP maintains a smooth, realistic gait while baseline models often collapse.
Discussion and Future Work
PhaseMP's primary innovation is treating frequency-domain features as a proxy for physical momentum. Because phase is unidirectional, it naturally prevents the "backward" glitches common in other temporal models. Furthermore, frequency-domain features remain more stable over time than raw joint positions, providing a reliable anchor for optimization.
Limitations: The current system assumes a flat ground plane and a static camera. Additionally, phase prediction can eventually drift during excessively long periods of total invisibility (several seconds).
Future Directions: Potential improvements include using phase features as positional encodings in Transformer architectures or integrating multi-modal signals, such as the audio of footsteps, to further constrain the phase estimation.
Summary updated