Application of Graph Convolutions in a Lightweight Model for Skeletal Human Motion Forecasting

Luca Hermes, Barbara Hammer, Malte Schilling

Introduction

Predicting human motion over short temporal horizons (typically several hundred milliseconds) is a foundational challenge for autonomous systems and safe human-robot collaboration. While Recurrent Neural Networks (RNNs) have historically dominated this field, they often treat skeletal joints as independent vectors, failing to exploit the inherent physical constraints of the human body. Recent shifts toward Geometric Deep Learning have introduced Graph Convolutional Networks (GCNs) to model these spatial relationships. However, many state-of-the-art GCN-based models are computationally expensive and parameter-dense, limiting their utility in real-time or resource-constrained environments. This paper presents a lightweight spatio-temporal model that leverages the body's kinematic hierarchy to achieve high predictive accuracy with a significantly reduced parameter footprint.

Proposed Spatio-Temporal Architecture

The proposed model is an adaptation of the Graph-WaveNet architecture, specifically optimized for skeletal data. The core of the system consists of $N$ stacked blocks designed to capture both temporal dynamics and spatial dependencies through two primary operations:

  1. Spatio-Temporal Convolution (ST-Conv): To model temporal flow, the architecture utilizes 2D convolutions that operate across a "kinematic chain." Rather than treating joints in isolation, the filters sample a joint alongside its parent and grandparent in the skeletal tree. This approach allows the model to learn how movement propagates from the torso to the extremities.
  2. Spatial Graph-Convolution (K-GCN): Spatial relationships are handled by decomposing the skeleton into three directed subgraphs: one directed toward the root (parents), one toward the extremities (children), and one for self-loops. This directed approach is more computationally efficient than using a full undirected adjacency matrix.
The proposed model architecture featuring stacked Graph-WaveNet blocks with ST-Conv and K-GCN layers.

To avoid mathematical singularities like gimbal lock, the model represents joint rotations using quaternions (4D vectors). The system operates autoregressively, using its own previous predictions to iteratively generate future frames.

Experimental Evaluation

The model was evaluated using the Human3.6M (H3.6M) dataset, a benchmark featuring seven subjects performing 15 diverse actions such as walking, eating, and smoking. Data was downsampled to 25Hz, with the model utilizing a "seed" sequence of 32 frames (1.28s) to forecast a target sequence of 10 frames (400ms). Performance was benchmarked against several baselines, including standard LSTMs and high-capacity models like DMGNN and DCT-GCN.

Results and Discussion

Parameter Efficiency

The most striking result is the model's efficiency. It utilizes only $4.46 \times 10^5$ parameters, making it approximately 140 times smaller than competing models like DMGNN ($6.26 \times 10^7$ parameters). This reduction is achieved by using the kinematic tree as a structural prior, which prevents the model from having to learn basic human anatomy from scratch.

Predictive Performance

Despite its small size, the model maintains competitive accuracy across the H3.6M dataset, specifically outperforming state-of-the-art models in complex actions like eating and walking together. Qualitative analysis of joint trajectories demonstrates that the model is particularly adept at capturing the "phase" or timing of movements, even when projecting beyond the standard 400ms window.

Qualitative results showing predicted vs. ground truth joint trajectories for various actions. The model effectively captures movement phases.

Stability and Limitations

Unlike original Graph-WaveNet implementations that often converged to static "zero-velocity" predictions, this adapted version successfully maintains active dynamics. However, the authors note that "finer details are smoothed out" in longer-term forecasts, as the network tends to predict an average, safe path when uncertainty increases.

Conclusion and Future Work

This research demonstrates that incorporating domain-specific priors—such as the kinematic hierarchy of the human skeleton—allows for the creation of highly efficient forecasting models. The lightweight nature of this architecture makes it an ideal candidate for deployment on edge devices or real-time robotic controllers. Future work will likely focus on reducing the "smoothing" effect in long-term predictions to better capture high-frequency motion details.

Summary updated

BibTeX
@inproceedings{hermes2021application,
  author = {Luca Hermes and Barbara Hammer and Malte Schilling},
  title = {{Application of Graph Convolutions in a Lightweight Model for Skeletal Human Motion Forecasting}},
  booktitle = {ESANN},
  year = {2021},
  url = {https://openreview.net/forum?id=_uqtr8XiXP},
}
Figure