Human Motion Anticipation with Symbolic Label

Julian Tanke, Andreas Weber, Juergen GallarXiv 2019 · arXiv

Introduction

Predicting human movement over several seconds is a fundamental challenge in computer vision and robotics. While current deep learning models excel at short-term forecasting (under 250ms), long-term predictions often suffer from "mean pose collapse," where the model generates a static, ghostly average of all possible futures. Researchers from the University of Bonn propose that the key to overcoming this is incorporating high-level intent. By predicting symbolic action labels (e.g., "walking," "sitting") before generating skeletal poses, their framework maintains realistic motion for horizons up to four seconds.

The Challenges of Long-Term Forecasting

Human motion synthesis faces two primary technical hurdles:

  1. The "Mean Pose" Trap: In the face of future uncertainty, models often minimize loss by averaging potential outcomes, resulting in a static, unnatural pose.
  2. Error Accumulation: Recursive models, which use their own predictions as future inputs, tend to accumulate small errors that snowball into physical impossibilities or "jitter" over time.

The authors argue these issues arise because models lack a global understanding of the action being performed. Without knowing if a person intends to sit or continue walking, the model cannot maintain temporal consistency.

Proposed Methodology

The proposed framework decomposes motion anticipation into a three-stage pipeline:

  1. Pose Label Predictor ($f_L$): A Temporal Convolutional Network (TCN) that classifies the observed past motion into a symbolic label.
  2. Label Forecasting ($f_F$): A Sequence-to-Sequence (Seq2Seq) module that predicts a sequence of future symbolic labels based on the current state.
  3. Motion Generation ($f_G$): A Gated Recurrent Unit (GRU) network that generates 3D skeletal poses. Crucially, this generator is conditioned on the forecasted future labels, allowing it to "plan" transitions (e.g., bending knees in anticipation of a "sitting" label).
The three-stage architecture: (1) Label Prediction, (2) Label Forecasting, and (3) Motion Generation conditioned on future intent.

Experimental Setup and Results

The model was evaluated on the Human 3.6M (H3.6M) dataset using both human-annotated fine-grained labels and unsupervised labels generated via clustering.

Quantitative Performance

On the Normalized Power Spectrum Similarity (NPSS) metric—which evaluates the naturalness of motion in the frequency domain—the symbolic approach significantly outperformed baselines like QuaterNet for long-term horizons (2-4 seconds).

Qualitative Impact of Symbolic Guiding

The use of "hard" one-hot labels was found to be more effective than soft probabilities, as it forces the model to commit to a specific, physically plausible motion path. As shown in the qualitative results, the model successfully transitions between complex actions without freezing.

Qualitative comparison showing how different predicted symbolic labels (icons) guide the generation of distinct, realistic skeletal sequences.

Conclusion

By anchoring 3D pose generation in high-level symbolic intent, this research provides a robust solution to the "mean pose" problem in long-term motion forecasting. While the model's performance is tied to the accuracy of the label forecast, the approach is highly complementary to other generative techniques like GANs. This work paves the way for more lifelike digital humans in virtual reality, gaming, and human-robot interaction.

Summary updated

Abstract

Anticipating human motion depends on two factors: the past motion and the person's intention. While the first factor has been extensively utilized to forecast short sequences of human motion, the second one remains elusive. In this work we approximate a person's intention via a symbolic representation, for example fine-grained action labels such as walking or sitting down. Forecasting a symbolic representation is much easier than forecasting the full body pose with its complex inter-dependencies. However, knowing the future actions makes forecasting human motion easier. We exploit this connection by first anticipating symbolic labels and then generate human motion, conditioned on the human motion input sequence as well as on the forecast labels. This allows the model to anticipate motion changes many steps ahead and adapt the poses accordingly. We achieve state-of-the-art results on short-term as well as on long-term human motion forecasting.
BibTeX
@misc{tanke2019human,
  author = {Julian Tanke and Andreas Weber and Juergen Gall},
  title = {{Human Motion Anticipation with Symbolic Label}},
  year = {2019},
  eprint = {1912.06079},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.06079},
}
Figure