Non-local Graph Convolutional Network for joint Activity Recognition and Motion Prediction

Dianhao Zhang, Ngo Anh Vien, Mien Van, Seán F. McLoone

Abstract

Researchers have developed a novel "two-birds-one-stone" AI model that simultaneously recognizes human activity and predicts future motion using 3D skeleton data. By combining Graph Convolutional Networks (GCNs) with a "non-local" attention mechanism, the system captures long-range patterns in human motion, outperforming standard sequence models and providing a more efficient path toward safe human-robot collaboration.

Introduction

For a robot to work safely alongside a human—a field known as Human-Robot Collaboration (HRC)—it must master two interwoven skills: Activity Recognition (understanding the current action, such as "picking up a box") and Motion Prediction (anticipating where the human’s hand will be in the next half-second). Traditionally, these tasks are treated independently, which is computationally expensive and ignores the fact that knowing what someone is doing helps predict where they are going. Standard neural networks often struggle with capturing long-term dependencies or the complex, non-linear structure of the human skeleton. This paper addresses these challenges through a unified framework.

Methodology

The proposed Non-local Graph Convolutional Network (NGC) framework consists of three main stages designed to process spatial and temporal data holistically:

  1. GCN-LSTM Encoder: The human skeleton is treated as a graph where joints are nodes and bones are edges. A GCN extracts spatial features from each frame, while a Bidirectional LSTM (Bi-LSTM) processes the temporal flow of the movement.
  2. Non-local GCN (NGC) Module: Inspired by self-attention mechanisms, this module calculates the relationship between hidden states across the entire observed sequence. This allows the model to "look back" at the beginning of an action to better predict its end, regardless of temporal distance.
  3. Task-Specific Decoders: The Motion Branch uses an LSTM-based decoder to predict future poses, while the Recognition Branch employs a CNN combined with Conditional Random Fields (CRFs) to classify the activity.
Overall architecture of the NGC framework showing the GCN-LSTM encoder and joint task branches.

The entire system is jointly trained using a weighted loss function, allowing the two tasks to inform and improve each other's feature representations.

Experiments and Results

The model was evaluated on three gold-standard datasets: Human 3.6M, NTU RGB+D, and CMU Mocap.

  • Action Recognition: On the NTU RGB+D dataset, the NGC-CRF model achieved 95.2% accuracy on Cross-Subject tests and 98.5% on Cross-View tests, outperforming state-of-the-art models like ST-GCN and DGNN.
  • Short-term Prediction: On Human 3.6M, while some models were slightly better at very short intervals (80ms), the NGC model became the top performer at the 400ms mark, proving its strength in capturing meaningful context.
  • Long-term Prediction: On the CMU Mocap dataset, the model substantially exceeded baselines when predicting poses up to 1000ms (1 second) into the future, a horizon typically difficult for recurrent networks.
Visual comparison of motion prediction results against baselines and ground truth over a 1000ms horizon.

Key Insights and Discussion

  • The Power of "Non-local": By using a non-local approach, the model can identify that a specific "wind-up" motion at the start of a sequence is highly correlated with a "release" motion much later, leading to more stable long-term predictions.
  • Synergy via Joint Training: A trade-off value (λ) of 0.4 between prediction and recognition loss yielded the best results, confirming that features learned for recognizing an action help predict its physical trajectory.
  • Skeleton as a Graph: Using GCNs respects the biological connections of the body, preventing the information loss that occurs when flattening skeleton data into simple coordinate lists.

Conclusion

The NGC model demonstrates that joint activity recognition and motion prediction can significantly improve performance in human motion analysis. While current training uses joint-scaling, future iterations could refine how different body parts interact. The ultimate goal is to integrate this "brain" into physical collaborative robot controllers to improve safety and reduce latency in real-world environments.

Summary updated

Abstract

3D skeleton-based motion prediction and activity recognition are two interwoven tasks in human behaviour analysis. In this work, we propose a motion context modeling methodology that provides a new way to combine the advantages of both graph convolutional neural networks and recurrent neural networks for joint human motion prediction and activity recognition. Our approach is based on using an LSTM encoder-decoder and a non-local feature extraction attention mechanism to model the spatial correlation of human skeleton data and temporal correlation among motion frames. The proposed network can easily include two output branches, one for Activity Recognition and one for Future Motion Prediction, which can be jointly trained for enhanced performance. Experimental results on Human 3.6M, CMU Mocap and NTU RGB-D datasets show that our proposed approach provides the best prediction capability among baseline LSTM-based methods, while achieving comparable performance to other state-of-the-art methods.
BibTeX
@inproceedings{zhang2021non,
  author = {Dianhao Zhang and Ngo Anh Vien and Mien Van and Seán F. McLoone},
  title = {{Non-local Graph Convolutional Network for joint Activity Recognition and Motion Prediction}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=Znt9jMM8o_w},
}
Figure