Motion Prediction using Trajectory Cues

Zhenguang Liu, Pengxiang Su, Shuang Wu, Xuanjing Shen, Haipeng Chen, Yanbin Hao, Meng Wang

Abstract

Predicting human motion is a cornerstone of autonomous systems, yet traditional methods operating in high-dimensional pose space often struggle with physical plausibility and long-term stability. This paper introduces MPT (Motion Prediction leveraging Trajectory cues), a framework that shifts the focus from absolute 3D coordinates to a smoother trajectory space. By modeling joint velocities and utilizing a semi-constrained Graph Convolutional Network (GCN) that integrates skeletal physics, biological priors, and learned implicit dependencies, MPT achieves a significant 19.1% average accuracy improvement over existing state-of-the-art methods across diverse human and animal datasets.

Introduction: The Pose Space Challenge

Predicting how a human or animal will move in the immediate future is critical for applications ranging from autonomous driving to human-robot interaction. Traditionally, researchers have modeled this problem in pose space, attempting to predict the 3D coordinates of every joint simultaneously. However, pose space is inherently high-dimensional and complex, often leading to models that produce "frozen" or physically impossible motions.

Furthermore, standard models typically only consider connections between physically attached joints (e.g., elbow to wrist). They often ignore "hidden" correlations, such as the natural synchronization between a left arm and right leg during walking, or the mirror symmetry of limbs during complex tasks. This lack of holistic understanding results in a loss of coordination over longer time horizons.

Methodology: Trajectory Representation and Semantic GCNs

The MPT framework addresses these challenges through two primary innovations:

1. Trajectory Representation

Instead of raw 3D coordinates, MPT decomposes motion into frame-wise velocities—specifically magnitude and orientation—and the final observed position. This approach aligns with the Lagrangian formulation of dynamical systems, where velocity represents kinetic energy and position represents potential energy. This representation is inherently smoother and more predictable than raw pose data, making it easier for neural networks to learn stable patterns.

2. Semantic Enriched GCN

To capture the complex relationships between joints, MPT employs a "semi-constrained" Graph Convolutional Network. Unlike standard graphs limited to skeletal bones, this GCN utilizes three distinct types of edges:

  • Skeletal: Physical connections based on the biological skeleton.
  • Motion Prior Knowledge: Hard-coded connections representing biological symmetries (e.g., left-right limb mirroring).
  • Learnt: Implicit dependencies discovered by the model during training, such as the correlation between a head tilt and a subsequent change in torso direction.
The MPT architecture, featuring the Semantic Enriched GCN module and the trajectory-to-3D prediction pipeline.

The architecture processes these features through a Gated Recurrent Unit (GRU) to manage temporal sequences, followed by a reconstruction block that translates predicted trajectories back into viewable 3D skeletons.

Experimental Setup and Datasets

The model was rigorously evaluated across four major benchmarks to ensure generalizability:

  • Human3.6M (H3.6M): A gold-standard dataset featuring 3.6 million 3D poses across 15 human actions.
  • CMU MoCap: A large-scale dataset covering diverse athletic and daily activities.
  • Fish and Mouse Datasets: To test cross-species applicability, the authors utilized 3D pose sequences of animals with varying kinematic chains (21 joints for fish, 5 for mice) and stochastic movement patterns.

Key Results and Discussion

Quantitatively, MPT demonstrates a clear lead over previous state-of-the-art (SOTA) methods. On the Human3.6M dataset, the model achieved a 17.5% accuracy improvement in average Mean Per Joint Position Error (MPJPE). Across all four datasets, the average improvement reached 19.1%.

Qualitatively, the model avoids the common pitfalls of "motion freezing" or joint dislocation. While baseline models often struggle to maintain realistic poses beyond 500ms, MPT produces coordinated, smooth motions that closely track ground truth data even at the 1,000ms mark.

Qualitative comparison of MPT against baseline models for 'Eating' and 'TalkingPhoto' actions, showing superior stability at 1000ms.

Conclusion and Future Directions

By shifting the perspective from absolute positions to trajectory cues, MPT provides a more robust foundation for motion prediction. The combination of physical constraints and learnable implicit dependencies allows the model to remain grounded in reality while capturing subtle nuances of movement.

While the "horizon effect" remains a challenge—where prediction accuracy naturally decays as the future becomes more uncertain—future work may involve hybridizing this trajectory-based approach with generative models like GANs or Diffusion models to better navigate the multi-modal nature of long-term human intent.

Summary updated

Abstract

Predicting human motion from a historical pose sequence is at the core of many applications in computer vision. Current state-of-the-art methods concentrate on learning motion contexts in the pose space, however, the high dimensionality and complex nature of human pose invoke inherent difficulties in extracting such contexts. In this paper, we instead advocate to model motion contexts in the joint trajectory space, as the trajectory of a joint is smooth, vectorial, and gives sufficient information to the model. Moreover, most existing methods consider only the dependencies between skeletal connected joints, disregarding prior knowledge and the hidden connections between geometrically separated joints. Motivated by this, we present a semi-constrained graph to explicitly encode skeletal connections and prior knowledge, while adaptively learn implicit dependencies between joints. We also explore the applications of our approach to a range of objects including human, fish, and mouse. Surprisingly, our method sets the new state-of-the-art performance on 4 different benchmark datasets, a remarkable highlight is that it achieves a 19.1% accuracy improvement over current state-of-the-art in average. To facilitate future research, we have released our code at https://github.com/Pose-Group/MPT.
BibTeX
@inproceedings{liu2021motion,
  author = {Zhenguang Liu and Pengxiang Su and Shuang Wu and Xuanjing Shen and Haipeng Chen and Yanbin Hao and Meng Wang},
  title = {{Motion Prediction using Trajectory Cues}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01305},
}
Figure