HP-GAN: Probabilistic 3D human motion prediction via GAN

Emad Barsoum, John R. Kender, Zicheng Liu

Abstract

Predicting human motion is a fundamental challenge in computer vision, complicated by the non-deterministic nature of future actions. Traditional deterministic models often fail by predicting a "mean pose" that lacks realism over long time horizons. We present HP-GAN (Human Pose GAN), a sequence-to-sequence generative framework that produces multiple, diverse, and physically plausible 3D human trajectories. By combining a Wasserstein GAN (WGAN-GP) with custom anatomical loss functions, HP-GAN ensures skeletal consistency and temporal smoothness while capturing the inherent stochasticity of human movement.

Introduction

Predicting how a human will move in the immediate future is critical for applications ranging from autonomous driving to human-robot interaction. However, human motion is inherently non-deterministic; a single initial pose could lead to various distinct actions, such as waving, pointing, or reaching.

Most traditional deep learning approaches treat this as a regression problem, typically optimized using Mean Squared Error ($L_2$). Because $L_2$ penalizes deviations from a single ground truth, these models tend to converge on the "average" of all possible futures. This results in "mean poses" that appear blurred, static, or physically impossible as the prediction horizon increases. HP-GAN addresses this by learning a probability density function of future poses, allowing for the generation of multiple plausible paths from a single input.

Methodology

HP-GAN utilizes a generative adversarial network architecture tailored specifically for skeletal sequences. The system is composed of several key components designed to balance diversity with physical realism.

Architecture

  • Generator: A sequence-to-sequence (Seq2Seq) model using Gated Recurrent Units (GRUs). It maps a sequence of past poses and a 128-dimensional random noise vector ($z$) to a predicted future sequence. By varying $z$, the model explores different branches of the motion probability distribution.
  • Critic & Discriminator: The framework employs a modified WGAN-GP (Wasserstein GAN with Gradient Penalty). The Critic measures the Wasserstein distance between real and generated motion distributions to provide a stable training signal, while a separate Discriminator assesses the absolute realism of sequences on a scale from 0 to 1.
The HP-GAN training framework, integrating the Generator, Discriminator, and Critic with custom anatomical losses.

Anatomical Constraints

To ensure that generated skeletons remain physically valid, the authors introduced two custom loss functions:

  1. Pose Gradient Loss ($L_{pg}$): This ensures temporal continuity by penalizing excessive joint displacement between consecutive frames.
  2. Bone Loss ($L_b$): This enforces constant bone lengths throughout the sequence, preventing unrealistic limb stretching or compression.

Experimental Setup

The model was rigorously evaluated on two benchmark datasets: NTURGB-D, a large-scale dataset containing over 56,000 clips with significant sensor noise, and Human3.6M, a high-fidelity dataset captured via professional motion capture systems. These datasets provide a robust test for the model's ability to handle both clean and noisy real-world data.

Key Results and Discussion

HP-GAN demonstrates a significant ability to generate long-term predictions (over 30 frames) that remain plausible. A key insight is that the incorporation of physics-inspired regularizers ($L_{pg}$ and $L_b$) was essential not only for realism but also for stabilizing the GAN training process and preventing model collapse.

Diverse future motion predictions generated from a single input sequence, with corresponding discriminator realism scores.

As shown in the visual results, the model successfully generates diverse trajectories for the same input sequence. For instance, given a person starting a throwing motion, HP-GAN can produce variations in height and speed. The Discriminator's output serves as a quality proxy, with a majority of generated sequences achieving a realism score greater than 50%.

Conclusion

HP-GAN successfully moves beyond deterministic "average" predictions to provide a probabilistic view of human motion. While the model shows great promise in generating diverse and realistic skeletons, future work remains in improving training stability and increasing the interpretability of the latent space $z$. Disentangling this space could allow for more granular control over specific motion attributes like speed or direction, further enhancing the utility of generative motion models.

Summary updated

BibTeX
@inproceedings{barsoum2018hp,
  author = {Emad Barsoum and John R. Kender and Zicheng Liu},
  title = {{HP-GAN: Probabilistic 3D human motion prediction via GAN}},
  booktitle = {Workshops @ CVPR},
  year = {2018},
  url = {https://openreview.net/forum?id=HyZEc6bd-H},
}
Figure