3D Human motion anticipation and classification

Emad Barsoum, John R. Kender, Zicheng LiuarXiv 2020 · arXiv

Abstract

Predicting future human motion is a complex, non-deterministic task where traditional regression models often fail by producing unrealistic, "averaged" results. This paper introduces a sequence-to-sequence Generative Adversarial Network (GAN) framework that generates multiple plausible future motions from a single past sequence. By incorporating physics-inspired loss functions—such as energy and bone length constraints—and a novel quality-assessment network, the model ensures generated motions are both diverse and physically realistic. Additionally, the features learned by the discriminator are shown to significantly accelerate and improve the performance of action recognition tasks, bridging the gap between motion anticipation and classification.

Introduction

Human motion prediction is inherently challenging due to its non-deterministic nature. A single initial movement can lead to various distinct outcomes. Traditional deterministic models struggle with this uncertainty, often yielding physically impossible or "blurry" motions that represent a statistical mean rather than a realistic action. This research addresses these challenges by developing a model capable of both anticipation (forecasting future poses) and classification (identifying current actions), ensuring that generated sequences adhere to the underlying "grammar" of human movement.

Methods

The proposed framework utilizes a modified Wasserstein GAN with Gradient Penalty (WGAN-GP) to model the distribution of possible future motions.

Overview of the GAN-based architecture for motion anticipation and classification, highlighting the generator, discriminator, and physics-inspired loss functions.

Architecture Components

  • The Generator ($G$): A sequence-to-sequence model using Gated Recurrent Units (GRUs). It maps a sequence of past poses and a random noise vector ($z$) to a predicted future sequence. By varying $z$, the model explores the space of plausible futures.
  • The Discriminator ($D$): A multi-layer network designed to distinguish between real and generated motion sequences. As shown in the detailed architecture below, it utilizes skip connections and dropout to learn robust features. These features serve as a foundation for downstream action recognition tasks.
Detailed architecture of the Discriminator network, featuring skip connections and multiple output heads for classification and quality assessment.
  • Quality-Assessment Network: A recurrent network with temporal attention that ranks generated sequences by their "realness," providing a mechanism to select the most plausible predictions.

Physics-Inspired Loss Functions

To ensure physical plausibility, the authors integrate several specialized loss terms:

  • Diversity Loss: Encourages the generator to produce distinct outputs for different noise vectors, preventing mode collapse.
  • Energy Loss: Minimizes the kinetic energy of the motion to prevent erratic, non-human movements.
  • Bone Loss: Enforces constant bone lengths over time, preventing anatomical distortions like "stretchy" limbs.

Datasets

The model was rigorously evaluated on two benchmark 3D human pose datasets:

  1. NTURGB-D: A large-scale, "noisy" dataset containing over 56,000 action clips captured via Microsoft Kinect v2.
  2. Human3.6M: A high-fidelity motion capture (MoCap) dataset featuring 3.6 million 3D human poses, providing a clean baseline for evaluation.

Results and Key Insights

The experimental results highlight the dual benefits of the GAN framework for both generation and recognition.

  • The Discriminator as a Teacher: A key finding is that the discriminator's learned features are highly effective for action recognition. When used to initialize a recognition network, the model reached peak accuracy in approximately 20 epochs—half the time required for a model trained from scratch (40 epochs).
  • Data Efficiency: The pre-trained features proved particularly valuable in low-data regimes. With only 25% of the training data, the pre-trained model achieved 46.71% accuracy, nearly matching the 48.64% accuracy of a scratch-built model using the full dataset.
  • Qualitative Realism: The inclusion of Energy and Bone losses resulted in smoother, more human-like transitions. The model successfully generated diverse, plausible futures (e.g., different ways of waving or sitting) from identical starting sequences.

Conclusion and Future Work

This work demonstrates that GANs can effectively model the uncertainty of human motion while learning features useful for classification. Future research will focus on providing semantic control over the generation process (e.g., specifying a particular action to predict) and exploring the use of the generator for synthetic data augmentation. Additionally, the authors plan to investigate "reverse mapping" to identify the specific noise vectors associated with particular motion styles, which could enhance motion clustering and analysis.

Summary updated

BibTeX
@misc{barsoum2020d,
  author = {Emad Barsoum and John R. Kender and Zicheng Liu},
  title = {{3D Human motion anticipation and classification}},
  year = {2020},
  eprint = {2012.15378},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.15378},
}
Figure