Skeleton-Based Action Recognition With Multi-Stream Adaptive Graph Convolutional Networks

Lei Shi, Yifan Zhang, Jian Cheng, Hanqing Lu

TL;DR

This paper introduces the Multi-Stream Adaptive Graph Convolutional Network (MS-AAGCN), a framework that moves away from rigid, manually defined skeleton structures for action recognition. By allowing the model to learn the optimal "graph" of the human body directly from data and incorporating second-order information (bones and motion), the authors achieved state-of-the-art performance on major benchmarks like NTU-RGBD and Kinetics-Skeleton.

What Problem Does This Paper Address?

Traditional action recognition often relies on RGB video, which is computationally heavy and sensitive to lighting or background clutter. Skeleton-based recognition—tracking the 2D or 3D coordinates of joints—is a robust alternative.

The previous gold standard, ST-GCN (Spatio-Temporal Graph Convolutional Network), treated the human skeleton as a fixed graph based on physical anatomy. However, the authors identify three major flaws in this "fixed anatomy" approach:

  1. Non-Physical Dependencies: For an action like "clapping," the relationship between the two hands is crucial, yet in a standard skeleton graph, the hands are many "hops" away from each other.
  2. Fixed Hierarchy: Deep learning models are hierarchical; the features needed in the first layer (simple joint positions) are different from the semantic features needed in the last layer. A fixed graph doesn't allow for this evolution.
  3. Data Diversity: Different actions might require different "connections." Wiping one's face requires a strong link between the hand and head, while jumping does not.

Methods

The authors propose three primary innovations to solve these issues:

1. Adaptive Graph Convolutional Layer (AGCL)

Instead of a fixed adjacency matrix, the AGCL uses a three-part graph structure:

  • The Global Graph ($B_k$): A learned graph topology that is optimized for the entire dataset. It starts with the physical skeleton but evolves during training.
  • The Individual Graph ($C_k$): A sample-specific graph generated on-the-fly using a "similarity" score (normalized embedded Gaussian) between joints in a specific video clip.
  • Gating Mechanism: A learned coefficient that balances how much the model relies on the global vs. the individual graph at different layers.

2. STC-Attention Module

The model incorporates a Spatial-Temporal-Channel attention mechanism. This allows the network to "focus" its computational effort on the most informative joints (Spatial), the most critical moments in a sequence (Temporal), and the most discriminative feature maps (Channel).

3. Multi-Stream Framework

Most previous models only looked at joint coordinates (first-order information). This paper argues that bones (the vector between two joints) and motion (the change in joints/bones over time) are equally important. They use a four-stream architecture (Joints, Bones, Joint Motion, Bone Motion) and fuse the results at the end.

Datasets

The model was evaluated on two massive, industry-standard datasets:

  • NTU-RGBD: Contains 56,000 action clips across 60 classes. It is tested using "Cross-Subject" (CS) and "Cross-View" (CV) benchmarks.
  • Kinetics-Skeleton: A skeletonized version of the Kinetics-400 dataset, containing roughly 300,000 video clips. This is significantly more challenging due to the "wild" nature of the source YouTube videos.

Key Insights and Ideas

  • Anatomy isn't Destiny: The "aha moment" is that the optimal graph for recognizing an action doesn't have to look like a human skeleton. The model learns to "connect" distant joints if their coordination is key to an action.
  • Top-Down Individuality: The authors found that individual (sample-specific) graphs are more important in the top layers of the network. Lower layers benefit from a more stable, global structure, while higher layers need the flexibility to model specific nuances of a unique movement.
  • Bones Matter: Treating bones as vectors (second-order info) provides a natural invariance to the absolute position of the person in the frame, making the model much more robust.

Key Results

The MS-AAGCN significantly outperformed the baseline ST-GCN and other concurrent methods:

  • NTU-RGBD: Achieved 90.0% (CS) and 96.2% (CV) accuracy, an improvement of ~8% over the baseline ST-GCN.
  • Kinetics-Skeleton: Achieved 37.8% Top-1 accuracy, outperforming ST-GCN by 7.1%.
  • RGB Fusion: When the authors combined their skeleton model with a standard RGB model (using a novel "pose-guided cropping" strategy), they reached a staggering 99.0% accuracy on the NTU-RGBD CV benchmark.

Limitations and Future Work

While highly effective, the model's multi-stream approach increases computational complexity, as four separate GCNs must be processed before fusion.

The authors suggest two main paths forward:

  1. End-to-End Integration: Currently, the skeleton is extracted by a separate pose estimator (like OpenPose). Integrating pose estimation and action recognition into a single, unified trainable framework could improve efficiency.
  2. Smarter Fusion: Exploring more sophisticated ways to fuse RGB and skeleton data beyond simple score summation.

Summary updated

Abstract

Graph convolutional networks (GCNs), which generalize CNNs to more generic non-Euclidean structures, have achieved remarkable performance for skeleton-based action recognition. However, there still exist several issues in the previous GCN-based models. First, the topology of the graph is set heuristically and fixed over all the model layers and input data. This may not be suitable for the hierarchy of the GCN model and the diversity of the data in action recognition tasks. Second, the second-order information of the skeleton data, i.e., the length and orientation of the bones, is rarely investigated, which is naturally more informative and discriminative for the human action recognition. In this work, we propose a novel multi-stream attention-enhanced adaptive graph convolutional neural network (MS-AAGCN) for skeleton-based action recognition. The graph topology in our model can be either uniformly or individually learned based on the input data in an end-to-end manner. This data-driven approach increases the flexibility of the model for graph construction and brings more generality to adapt to various data samples. Besides, the proposed adaptive graph convolutional layer is further enhanced by a spatial-temporal-channel attention module, which helps the model pay more attention to important joints, frames and features. Moreover, the information of both the joints and bones, together with their motion information, are simultaneously modeled in a multi-stream framework, which shows notable improvement for the recognition accuracy. Extensive experiments on the two large-scale datasets, NTU-RGBD and Kinetics-Skeleton, demonstrate that the performance of our model exceeds the state-of-the-art with a significant margin.
BibTeX
@article{shi2020skeleton,
  author = {Lei Shi and Yifan Zhang and Jian Cheng and Hanqing Lu},
  title = {{Skeleton-Based Action Recognition With Multi-Stream Adaptive Graph Convolutional Networks}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=42MytsHdhV},
}

Loading related content...

Figure