Geometry-Aware Deep Learning for 3D Skeleton-Based Motion Prediction

Mayssa Zaier, Hazem Wannous, Hassen DriraWACV 2025

TL;DR

This paper introduces MAN-TF, a novel deep learning architecture for 3D skeleton-based human motion prediction. By representing human motion using both the geometric properties of Kendall's shape space (for non-rigid deformations) and Lie groups (for rigid transformations), MAN-TF achieves superior long-term prediction accuracy compared to existing methods.

What Problem Does This Paper Address?

Predicting future human motion from sequences of 3D skeleton data is crucial for applications like human-robot interaction and autonomous driving. However, existing deep learning models often struggle with the inherent non-linear dynamics and high dimensionality of human movement. Specifically, conventional Recurrent Neural Networks (RNNs) fail to capture long-term dependencies, leading to unnatural and abrupt predicted poses. While some methods attempt to address spatial correlations, they often ignore the intricate relationships within joints. This paper tackles the challenge of accurately predicting future poses by developing a representation that captures both the intrinsic shape of the skeleton and its rigid transformations over time, aiming for smoother and more coherent long-term predictions.

Methods

The core innovation of this paper lies in its geometry-aware representation of 3D human skeletons. Instead of treating skeleton data as points in a standard Euclidean space, the authors leverage two manifold-valued representations:

  • Kendall's Shape Space: This representation captures the intrinsic shape of the skeleton, independent of rigid transformations like rotation and translation. It models non-rigid deformations, such as the bending of limbs.
  • Lie Group (SE(3)) and Lie Algebra (se(3)): These representations model the rigid transformations (rotation and translation) of the entire skeleton relative to a reference pose.

These two representations are fed into a Transformer-based architecture with a bimodal encoder-decoder structure. The model uses:

  • Separate Encoder Branches: One for Kendall's shape space data and another for Lie group/algebra data.
  • Self-Attention Mechanisms: Within each branch to capture temporal dependencies.
  • Cross-Attention Mechanism: Crucially, this mechanism allows the model to learn the interplay and dependencies between the rigid (Lie group) and non-rigid (Kendall's shape space) transformations. This is vital because internal body movements often precede and influence global body rotations.
  • Logarithmic Mapping: To map data from the manifold to a tangent space for easier processing by the Transformer.

The paper also proposes tailored loss functions: L2 loss for the Kendall's shape space representation (emphasizing spatial relationships) and a Weighted Loss of Angles for the Lie group representation (focusing on rotational aspects).

Datasets

The proposed method is evaluated on three widely used benchmarks:

  • H3.6m: A large-scale dataset with 3.6 million 3D human poses across 15 activities.
  • GTA-IM: A synthetic dataset featuring indoor environments and diverse characters performing various activities.
  • PROX: A dataset captured with a Kinect-One sensor, including human body skeletons, RGB sequences, and 3D scenes with interacting subjects.

Key Insights and Ideas

  • Manifold Representations are Key: The paper strongly advocates for moving beyond Euclidean representations for human motion. The non-linear nature of human poses necessitates geometric approaches.
  • Synergy of Rigid and Non-Rigid: The most significant insight is that combining representations for both rigid (Lie group) and non-rigid (Kendall's shape space) transformations leads to superior prediction accuracy. Neither representation alone is sufficient.
  • Cross-Attention for Interplay: The cross-attention mechanism is crucial for learning how internal body movements (non-rigid) influence global body transformations (rigid), which is essential for realistic motion prediction.
  • Transformer's Suitability: Transformers, with their attention mechanisms, are well-suited to capture long-range dependencies and complex relationships between different parts of the skeleton and across time, outperforming traditional RNNs in this regard.

Key Results

The MAN-TF model demonstrates state-of-the-art performance on the H3.6m dataset, achieving lower Mean Angle Error (MAE) and Mean Per Joint Position Error (MPJPE) compared to numerous existing methods, especially for long-term predictions. Ablation studies confirm that the combined use of Lie group and Kendall's shape space representations significantly outperforms using either representation alone or a standard Euclidean space representation. On the GTA-IM and PROX datasets, MAN-TF also shows superiority in predicting both 3D paths and poses.

Limitations and Future Work

The authors acknowledge that while their model excels in pose prediction, it exhibits slightly higher error rates in 3D path prediction on the GTA-IM dataset, suggesting a potential trade-off between pose and path objectives, possibly due to the absence of visual cues.

Future work directions include:

  • Incorporating External Factors: Considering environmental context and the motion of other people for more accurate predictions.
  • Multi-Skeleton Motion Prediction: Extending the approach to predict the interactions and coordinated movements of multiple individuals.
  • Enhancing Kendall Shape Parameters: Further research into optimizing the preprocessing of Kendall shape parameters.
  • Broader Dataset Evaluation: Evaluating the model on a wider range of diverse datasets capturing varied behaviors and environments.

Summary updated

BibTeX
@inproceedings{zaier2025geometry,
  author = {Mayssa Zaier and Hazem Wannous and Hassen Drira},
  title = {{Geometry-Aware Deep Learning for 3D Skeleton-Based Motion Prediction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure