Abstract
Capturing realistic human motion from video remains challenging due to noise, occlusions, and physical implausibility. We present a Hierarchical Motion Variational Autoencoder (HM-VAE) that serves as a task-generic prior for human movement. By learning a structured latent space from extensive motion-capture data, the HM-VAE can be applied to diverse tasks such as pose refinement, motion completion, and interpolation without task-specific training. Our approach leverages a skeleton-aware architecture and a two-level hierarchical latent space to capture both coarse global movements and fine-grained local poses.
Introduction
Human motion capture from monocular video often suffers from temporal jitter and anatomical inaccuracies. Traditional methods address these issues using task-specific models tailored for denoising, interpolation, or prediction. This paper proposes a "universal prior"—a deep generative model that understands the manifold of valid human motion. Once trained, this prior can be integrated into various downstream applications to ensure physical plausibility and temporal smoothness.
Methodology
The core of our approach is the Hierarchical Motion Variational Autoencoder (HM-VAE). Unlike standard VAEs, HM-VAE utilizes a two-level latent representation:
- Global Latent Space: Encodes the overall coarse movement of the body.
- Local Latent Space: Captures fine-grained details and specific joint configurations.
Skeleton-Aware Architecture
To respect the kinematic constraints of the human body, the model employs specialized skeleton-aware convolutions and pooling. These operations follow the skeletal hierarchy, ensuring that the model understands joint dependencies (e.g., the relationship between the shoulder and elbow) rather than treating joints as independent data points.
Temporal Processing and Trajectory Prediction
The model processes motion in overlapping windows using a sliding window strategy to ensure temporal consistency across sequences of arbitrary length. Additionally, a dedicated trajectory module estimates root velocity, allowing for the reconstruction of motion in world coordinates. For specific tasks like motion completion, a two-phase optimization process is used: first optimizing latent codes to match observations, followed by fine-tuning the decoder for a precise fit.
Experimental Results
The HM-VAE was trained on the AMASS dataset and evaluated on 3DPW and LAFAN1.
Pose Refinement and Denoising
When used to refine outputs from state-of-the-art models like VIBE, our method significantly reduces "jitter." On the 3DPW dataset, the acceleration error—a proxy for temporal instability—was reduced by over 75%, dropping from 27.12 $mm/s^2$ to 6.03 $mm/s^2$.
Motion Interpolation and Ablation
In motion interpolation tasks, the HM-VAE outperformed standard mathematical baselines like Slerp, particularly for long gaps (30-45 frames). Ablation studies confirmed that the hierarchical structure (HM-VAE) provides a 25% improvement in reconstruction accuracy over a flat latent space (M-VAE), highlighting the importance of separating global and local motion features.
Discussion and Conclusion
The HM-VAE demonstrates the power of a task-generic motion prior. By embedding skeletal constraints and hierarchical latent spaces, the model effectively filters noise and completes missing data while maintaining realism.
However, limitations remain. Long-sequence trajectory estimation can suffer from cumulative drift, and the lack of explicit physics-based constraints (like foot-contact losses) can occasionally lead to "foot sliding." Future work will focus on incorporating rigid physical properties and exploring few-shot adaptation for specialized motion domains.
Summary updated