STTG-net: a Spatio-temporal network for human motion prediction based on transformer and graph convolution network

Lujing Chen, Rui Liu, Xin Yang, Dongsheng Zhou, Qiang Zhang, Xiaopeng WeiVis. Comput. Ind. Biomed. Art 2022 · OpenReview

TL;DR

This paper introduces STTG-Net, a novel spatio-temporal network for human motion prediction that combines transformers for global temporal dependencies and Graph Convolutional Networks (GCNs) for local spatial joint correlations. To combat error accumulation and jerky predictions, they also propose a prediction revision module that fuses current and past predictions, leading to smoother and more accurate future motion forecasts.

What Problem Does This Paper Address?

Predicting future human motion from observed sequences is a challenging task due to the inherent complexity and stochastic nature of human movement. While existing methods often focus on capturing temporal dependencies between frames, they tend to overlook the crucial spatial relationships between different body joints. This oversight can lead to unrealistic motion predictions, characterized by accumulating errors and abrupt, unnatural movements, especially over longer prediction horizons. The paper aims to bridge this gap by explicitly modeling both the temporal evolution of motion and the spatial interactions between body parts.

Methods

The core of STTG-Net is its hybrid architecture:

  • Temporal Transformer (T-transformer): This component leverages the power of transformers to capture long-range temporal dependencies within the motion sequence. It treats each human pose as a "word" and uses multi-head self-attention to understand how poses evolve over time globally.
  • Spatial Graph Convolutional Network (S-GCN): To address the spatial relationships, an S-GCN module is employed. It models the human skeleton as a graph, where joints are nodes and connections are edges. This allows the network to learn local spatial correlations between joints within each frame.
  • Discrete Cosine Transform (DCT): Before feeding data into the T-transformer, DCT is used to encode motion information into a more compact trajectory space, discarding high-frequency signals to capture smoothness. An Inverse DCT (IDCT) is then used to recover the predicted pose.
  • Prediction Revision Module: To mitigate error accumulation and improve motion smoothness, a novel fusion-based revision module is introduced. This module takes the current frame's prediction and fuses it with the previous frame's "final prediction" using learned coefficients (α and β). This helps to "pull back" predictions that deviate too much, preventing sudden jumps and ensuring continuity.

Datasets

The primary dataset used for training and evaluation is Human3.6M. This is a widely used, large-scale dataset containing 3D human pose data from motion capture, along with corresponding RGB images, depth images, and body surface data. It covers 15 different actions performed by seven subjects. The authors followed standard data processing practices, downsampling to 25 frames per second and using specific subjects for training, validation, and testing.

Key Insights and Ideas

  • Spatio-temporal coupling is key: The paper strongly emphasizes that effective human motion prediction requires modeling both temporal dynamics and spatial joint interactions, not just one in isolation.
  • Transformers for global temporal context: The success of transformers in other vision tasks is leveraged to capture long-range temporal dependencies, which is often a weakness of traditional RNN-based approaches.
  • GCNs for local spatial structure: Graph convolutional networks are a natural fit for modeling the interconnected structure of the human skeleton, allowing for the capture of how joints influence each other locally.
  • Fusion for smoother predictions: The prediction revision module offers a simple yet effective way to enforce motion continuity by blending current predictions with past results, directly addressing the issue of jerky motion.
  • DCT as an efficient representation: Using DCT to compact motion data before temporal modeling is an interesting choice that aids in capturing smoothness.

Key Results

  • STTG-Net achieved state-of-the-art results on more than half of the actions on the Human3.6M dataset, outperforming several strong baselines, including recent transformer-based methods.
  • The proposed method demonstrated reduced prediction error and smoother motion compared to previous methods.
  • Specifically, STTG-Net achieved better average error results than the ST-transformer [34] method, particularly in short-term forecasting.
  • The prediction error of STTG-Net grew slower with increasing prediction time, indicating less error accumulation.
  • The model is relatively lightweight, with a total parameter count of 2.33M, significantly less than some comparable methods (e.g., 3.08M for ref. [27]).
  • Ablation studies confirmed the effectiveness of each component: the T-transformer, S-GCN, and the prediction revision module all contributed significantly to the overall performance.

Limitations and Future Work

  • Long-term prediction: While STTG-Net excels in short-term prediction, the paper acknowledges that for longer-term predictions (beyond 1s), spatio-temporal transformers might offer more advantages.
  • Parameter reduction: The authors aim to further reduce the model's parameter count and explore algorithms for learning more sophisticated fusion strategies within the correction modules.
  • Longer-term motion prediction: Future work will focus on developing models capable of more accurate and robust predictions for extended future motion sequences.
  • Fusion coefficients: While specific coefficients (α=0.125, β=0.875) were found to yield the best results, the paper notes that the revision module can sometimes introduce minor errors for predictions with already small errors.

Summary updated

Abstract

In recent years, human motion prediction has become an active research topic in computer vision. However, owing to the complexity and stochastic nature of human motion, it remains a challenging problem. In previous works, human motion prediction has always been treated as a typical inter-sequence problem, and most works have aimed to capture the temporal dependence between successive frames. However, although these approaches focused on the effects of the temporal dimension, they rarely considered the correlation between different joints in space. Thus, the spatio-temporal coupling of human joints is considered, to propose a novel spatio-temporal network based on a transformer and a gragh convolutional network (GCN) (STTG-Net). The temporal transformer is used to capture the global temporal dependencies, and the spatial GCN module is used to establish local spatial correlations between the joints for each frame. To overcome the problems of error accumulation and discontinuity in the motion prediction, a revision method based on fusion strategy is also proposed, in which the current prediction frame is fused with the previous frame. The experimental results show that the proposed prediction method has less prediction error and the prediction motion is smoother than previous prediction methods. The effectiveness of the proposed method is also demonstrated comparing it with the state-of-the-art method on the Human3.6 M dataset.
BibTeX
@inproceedings{chen2022sttg,
  author = {Lujing Chen and Rui Liu and Xin Yang and Dongsheng Zhou and Qiang Zhang and Xiaopeng Wei},
  title = {{STTG-net: a Spatio-temporal network for human motion prediction based on transformer and graph convolution network}},
  booktitle = {Vis. Comput. Ind. Biomed. Art},
  year = {2022},
  url = {https://openreview.net/forum?id=lFaZvgovwq},
}
Figure