Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks

Lei Shi, Yifan Zhang, Jian Cheng, Hanqing Lu

Abstract

Graph convolutional networks (GCNs), which generalize CNNs to more generic non-Euclidean structures, have achieved remarkable performance for skeleton-based action recognition. However, there still exist several issues in the previous GCN-based models. First, the topology of the graph is set heuristically and fixed over all the model layers and input data. This may not be suitable for the hierarchy of the GCN model and the diversity of the data in action recognition tasks. Second, the second-order information of the skeleton data, i.e., the length and orientation of the bones, is rarely investigated, which is naturally more informative and discriminative for the human action recognition. In this work, we propose a novel multi-stream attention-enhanced adaptive graph convolutional neural network (MS-AAGCN) for skeleton-based action recognition. The graph topology in our model can be either uniformly or individually learned based on the input data in an end-to-end manner. This data-driven approach increases the flexibility of the model for graph construction and brings more generality to adapt to various data samples. Besides, the proposed adaptive graph convolutional layer is further enhanced by a spatial-temporal-channel attention module, which helps the model pay more attention to important joints, frames and features. Moreover, the information of both the joints and bones, together with their motion information, are simultaneously modeled in a multi-stream framework, which shows notable improvement for the recognition accuracy. Extensive experiments on the two large-scale datasets, NTU-RGBD and Kinetics-Skeleton, demonstrate that the performance of our model exceeds the state-of-the-art with a significant margin.

Introduction

Skeleton-based action recognition, which utilizes the 2D or 3D coordinates of human joints, has emerged as a robust alternative to traditional RGB-based methods. Unlike video-based recognition, skeleton data is computationally efficient and resilient to changes in lighting or background clutter. The previous state-of-the-art, the Spatio-Temporal Graph Convolutional Network (ST-GCN), treated the human skeleton as a fixed graph based on physical anatomy. However, this approach has significant drawbacks: it cannot easily capture dependencies between non-adjacent joints (e.g., hands during a clapping motion), it uses a fixed hierarchy that doesn't adapt to deep feature evolution, and it lacks the flexibility to handle the diversity of different actions.

Methodology

Adaptive Graph Convolutional Layer (AGCL)

To address the limitations of fixed topologies, we propose the Adaptive Graph Convolutional Layer (AGCL). Instead of a static adjacency matrix, the AGCL employs a three-part graph structure that allows the network to learn the optimal connections directly from the data. This includes a learned global graph ($B_k$) that captures dataset-wide patterns and an individual graph ($C_k$) that is generated on-the-fly for each specific sample using a similarity score between joints. A gating mechanism balances these components, allowing the model to adapt its connectivity at different layers of the network.

Architecture of the Adaptive Graph Convolutional Layer (AGCL), illustrating the integration of global and individual learned graph structures.

STC-Attention Module

The model is further enhanced by a Spatial-Temporal-Channel (STC) attention module. This component allows the network to focus its computational resources on the most informative joints (Spatial), critical moments in time (Temporal), and discriminative feature maps (Channel). By sequentially refining the features through these three dimensions, the model becomes more robust to noise and irrelevant joint movements.

The Spatial-Temporal-Channel (STC) Attention Module, which sequentially refines input features to focus on key joints, frames, and channels.

Multi-Stream Framework

Beyond joint coordinates (first-order information), we incorporate second-order information by modeling bones (the vectors between joints) and motion (the temporal change in joints and bones). We utilize a four-stream architecture—Joints, Bones, Joint Motion, and Bone Motion—and fuse their outputs at the final stage. This multi-stream approach provides a more holistic view of human movement, significantly improving recognition accuracy.

Experiments and Results

We evaluated MS-AAGCN on two major benchmarks: NTU-RGBD and Kinetics-Skeleton.

On the NTU-RGBD dataset, our model achieved 90.0% accuracy on the Cross-Subject (CS) benchmark and 96.2% on the Cross-View (CV) benchmark, outperforming the ST-GCN baseline by approximately 8%. When combined with a standard RGB model using a pose-guided cropping strategy, the accuracy reached a staggering 99.0% on the CV benchmark.

On the Kinetics-Skeleton dataset, which is derived from unconstrained YouTube videos, MS-AAGCN achieved 37.8% Top-1 accuracy, a 7.1% improvement over ST-GCN. Our analysis showed that the learned individual graphs are particularly beneficial in the top layers of the network, where high-level semantic features require more flexibility to distinguish between complex actions.

Conclusion

The MS-AAGCN framework demonstrates that data-driven, adaptive graph topologies are superior to fixed anatomical structures for action recognition. By integrating multi-stream second-order information and a comprehensive attention mechanism, we achieve state-of-the-art performance on large-scale datasets. Future research will explore end-to-end integration with pose estimation and more advanced multi-modal fusion strategies.

Summary updated

Abstract

Graph convolutional networks (GCNs), which generalize CNNs to more generic non-Euclidean structures, have achieved remarkable performance for skeleton-based action recognition. However, there still exist several issues in the previous GCN-based models. First, the topology of the graph is set heuristically and fixed over all the model layers and input data. This may not be suitable for the hierarchy of the GCN model and the diversity of the data in action recognition tasks. Second, the second-order information of the skeleton data, i.e., the length and orientation of the bones, is rarely investigated, which is naturally more informative and discriminative for the human action recognition. In this work, we propose a novel multi-stream attention-enhanced adaptive graph convolutional neural network (MS-AAGCN) for skeleton-based action recognition. The graph topology in our model can be either uniformly or individually learned based on the input data in an end-to-end manner. This data-driven approach increases the flexibility of the model for graph construction and brings more generality to adapt to various data samples. Besides, the proposed adaptive graph convolutional layer is further enhanced by a spatial-temporal-channel attention module, which helps the model pay more attention to important joints, frames and features. Moreover, the information of both the joints and bones, together with their motion information, are simultaneously modeled in a multi-stream framework, which shows notable improvement for the recognition accuracy. Extensive experiments on the two large-scale datasets, NTU-RGBD and Kinetics-Skeleton, demonstrate that the performance of our model exceeds the state-of-the-art with a significant margin.
BibTeX
@article{shi2020skeleton,
  author = {Lei Shi and Yifan Zhang and Jian Cheng and Hanqing Lu},
  title = {{Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=42MytsHdhV},
}
Figure