Human Motion Prediction via Dual-Attention and Multi-Granularity Temporal Convolutional Networks

Biaozhang Huang, Xinde Li2023 · DOI

TL;DR

This paper introduces DA-MgTCNs, a novel deep learning architecture for human motion prediction that significantly improves accuracy by jointly leveraging dual-attention mechanisms for spatial feature extraction and multi-granularity temporal convolutional networks for capturing complex temporal dependencies. The proposed method achieves state-of-the-art results on benchmark datasets, demonstrating its effectiveness in predicting both short-term and long-term human movements.

What Problem Does This Paper Address?

Accurately predicting human motion is crucial for enabling intelligent devices to interact seamlessly and harmoniously with humans. Existing methods often struggle to fully capture the intricate spatial relationships between body joints and the complex temporal dynamics inherent in motion sequences. This limitation leads to unsatisfactory predictions, especially for longer time horizons or more irregular movements. The paper aims to bridge this gap by developing a model that can effectively mine both spatial and temporal information at multiple scales.

Methods

The core of the proposed solution is the DA-MgTCN module, which combines two key innovations:

  • Dual-Attention (DA) Module: This module extracts spatial features by employing both joint-level attention and channel-level attention. Joint-level attention focuses on the relationships between different body joints, while channel-level attention considers the correlations within the 3D coordinate dimensions (x, y, z) of each joint. This dual approach allows for a richer understanding of spatial configurations.
  • Multi-Granularity Temporal Convolutional Network (MgTCN) Module: Building upon Temporal Convolutional Networks (TCNs), which are known for their ability to capture long-term dependencies, the MgTCN introduces multi-granularity convolutions. This means the network uses convolutional kernels of varying sizes (receptive fields) across different branches. This allows the model to flexibly capture temporal dependencies at multiple scales, from short-term nuances to long-term trends, which is crucial for complex human movements.

The overall architecture uses a residual depth network composed of these DA-MgTCN modules, trained end-to-end. Global and local residual connections are incorporated to facilitate gradient flow and improve training stability in deeper networks.

Datasets

The proposed method was evaluated on two widely used large-scale human motion capture datasets:

  • Human3.6M: This dataset contains 3D human poses for 7 actors performing 15 different actions. The authors used 17 joints per pose and downsampled the frame rate to 25 fps. Actors S5 and S11 were used for testing and validation, while the remaining actors were used for training.
  • CMU-MoCap: This dataset, collected using 12 Vicon cameras, describes human motion with 41 sensors. The authors preprocessed the data similarly to Human3.6M, simplifying poses and reducing the frame rate to 25 fps. Eight specific actions were selected for evaluation.

Key Insights and Ideas

  • Spatial correlations are multi-faceted: The paper highlights that simply considering joint relationships isn't enough. Incorporating attention across the 3D coordinate dimensions (x, y, z) of each joint provides crucial complementary spatial information.
  • Temporal dependencies are scale-dependent: Human motion isn't just a series of discrete steps; it involves patterns at various time scales. The multi-granularity approach in TCNs is key to capturing both the immediate actions and the overarching trends.
  • Combining spatial and temporal intelligence is synergistic: The DA-MgTCN architecture demonstrates that effectively integrating sophisticated spatial feature extraction with flexible temporal modeling leads to superior prediction performance.
  • Robustness to complex motions: The method shows particular strength in predicting more irregular and non-periodic movements, which are often challenging for existing models.

Key Results

The DA-MgTCN model achieved state-of-the-art performance on both the Human3.6M and CMU-MoCap datasets.

  • Quantitative: The paper reports significant improvements in Mean Per Joint Position Error (MPJPE) across various prediction horizons (80 ms to 1000 ms). For instance, on Human3.6M, the model achieved the best MPJPE for the 560 ms and 1000 ms predictions. On CMU-MoCap, the DA-MgTCN improved average prediction accuracy by approximately 1.5% in short-term and 3% in long-term predictions compared to the state-of-the-art ST-DGCN method.
  • Qualitative: Visualizations showed that the DA-MgTCN produced predictions with higher similarity to ground truth, exhibiting lower distortion and better continuity. The model demonstrated a more coordinated and coherent representation of motion, particularly in complex actions.

Limitations and Future Work

The authors acknowledge that predicting very long-term future poses remains challenging due to the inherent uncertainty in human motion and the difficulty in capturing long-term dependencies between joints and frames. They also note that experiments were conducted under specific realistic data scenarios, which might have posed challenges.

Future work will focus on:

  • Further optimizing the network structure and parameter settings.
  • Extending the model's application to spatio-temporal prediction tasks in real-world scenarios, such as robot perception and interaction.
  • Investigating novel methods for multi-grain human motion prediction in more intricate scenarios and multi-domain contexts to enhance adaptability and performance.

Summary updated

BibTeX
@misc{huang2023human,
  author = {Biaozhang Huang and Xinde Li},
  title = {{Human Motion Prediction via Dual-Attention and Multi-Granularity Temporal Convolutional Networks}},
  year = {2023},
  doi = {10.3390/s23125653},
}
Figure