Task-Generic Hierarchical Human Motion Prior using VAEs

Jiaman Li, Ruben Villegas, Duygu Ceylan, Jimei Yang, Zhengfei Kuang, Hao Li, Yajie Zhao

Abstract

Capturing realistic human motion from video remains challenging due to noise, occlusions, and physical implausibility. We present a Hierarchical Motion Variational Autoencoder (HM-VAE) that serves as a task-generic prior for human movement. By learning a structured latent space from extensive motion-capture data, the HM-VAE can be applied to diverse tasks such as pose refinement, motion completion, and interpolation without task-specific training. Our approach leverages a skeleton-aware architecture and a two-level hierarchical latent space to capture both coarse global movements and fine-grained local poses.

Introduction

Human motion capture from monocular video often suffers from temporal jitter and anatomical inaccuracies. Traditional methods address these issues using task-specific models tailored for denoising, interpolation, or prediction. This paper proposes a "universal prior"—a deep generative model that understands the manifold of valid human motion. Once trained, this prior can be integrated into various downstream applications to ensure physical plausibility and temporal smoothness.

The HM-VAE acts as a universal prior, enabling various motion tasks like interpolation and refinement through a learned latent space.

Methodology

The core of our approach is the Hierarchical Motion Variational Autoencoder (HM-VAE). Unlike standard VAEs, HM-VAE utilizes a two-level latent representation:

  1. Global Latent Space: Encodes the overall coarse movement of the body.
  2. Local Latent Space: Captures fine-grained details and specific joint configurations.

Skeleton-Aware Architecture

To respect the kinematic constraints of the human body, the model employs specialized skeleton-aware convolutions and pooling. These operations follow the skeletal hierarchy, ensuring that the model understands joint dependencies (e.g., the relationship between the shoulder and elbow) rather than treating joints as independent data points.

Temporal Processing and Trajectory Prediction

The model processes motion in overlapping windows using a sliding window strategy to ensure temporal consistency across sequences of arbitrary length. Additionally, a dedicated trajectory module estimates root velocity, allowing for the reconstruction of motion in world coordinates. For specific tasks like motion completion, a two-phase optimization process is used: first optimizing latent codes to match observations, followed by fine-tuning the decoder for a precise fit.

Experimental Results

The HM-VAE was trained on the AMASS dataset and evaluated on 3DPW and LAFAN1.

Pose Refinement and Denoising

When used to refine outputs from state-of-the-art models like VIBE, our method significantly reduces "jitter." On the 3DPW dataset, the acceleration error—a proxy for temporal instability—was reduced by over 75%, dropping from 27.12 $mm/s^2$ to 6.03 $mm/s^2$.

Quantitative and qualitative comparison of motion refinement. The proposed method significantly reduces acceleration error (jitter) compared to the VIBE baseline.

Motion Interpolation and Ablation

In motion interpolation tasks, the HM-VAE outperformed standard mathematical baselines like Slerp, particularly for long gaps (30-45 frames). Ablation studies confirmed that the hierarchical structure (HM-VAE) provides a 25% improvement in reconstruction accuracy over a flat latent space (M-VAE), highlighting the importance of separating global and local motion features.

Discussion and Conclusion

The HM-VAE demonstrates the power of a task-generic motion prior. By embedding skeletal constraints and hierarchical latent spaces, the model effectively filters noise and completes missing data while maintaining realism.

However, limitations remain. Long-sequence trajectory estimation can suffer from cumulative drift, and the lack of explicit physics-based constraints (like foot-contact losses) can occasionally lead to "foot sliding." Future work will focus on incorporating rigid physical properties and exploring few-shot adaptation for specialized motion domains.

Summary updated

Abstract

A deep generative model that describes human motions can benefit a wide range of fundamental computer vision and graphics tasks, such as providing robustness to video-based human pose estimation, predicting complete body movements for motion capture systems during occlusions, and assisting key frame animation with plausible movements. In this paper, we present a method for learning complex human motions independent of specific tasks using a combined global and local latent space to facilitate coarse and fine-grained modeling. Specifically, we propose a hierarchical motion variational autoencoder (HM-VAE) that consists of a 2-level hierarchical latent space. While the global latent space captures the overall global body motion, the local latent space enables to capture the refined poses of the different body parts. We demonstrate the effectiveness of our hierarchical motion variational autoencoder in a variety of tasks including video-based human pose estimation, motion completion from partial observations, and motion synthesis from sparse key-frames. Even though, our model has not been trained for any of these tasks specifically, it provides superior performance than task-specific alternatives. Our general-purpose human motion prior model can fix corrupted human body animations and generate complete movements from incomplete observations.
BibTeX
@inproceedings{li2021task,
  author = {Jiaman Li and Ruben Villegas and Duygu Ceylan and Jimei Yang and Zhengfei Kuang and Hao Li and Yajie Zhao},
  title = {{Task-Generic Hierarchical Human Motion Prior using VAEs}},
  booktitle = {3DV},
  year = {2021},
  url = {https://openreview.net/forum?id=Dc6jEXGMT71},
}
Figure