Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition

Lei Shi, Yifan Zhang, Jian Cheng, Hanqing LuOpenReview 2025 · OpenReview

TL;DR

This paper introduces the Two-Stream Adaptive Graph Convolutional Network (2s-AGCN) for skeleton-based action recognition. It addresses limitations of fixed graph structures in existing methods by learning adaptive graph topologies and incorporates crucial "second-order" bone information alongside joint data for improved performance.

What Problem Does This Paper Address?

Skeleton-based action recognition aims to understand human actions by analyzing the movements of body joints over time. Graph Convolutional Networks (GCNs) have emerged as a powerful tool for this, naturally modeling skeletons as graphs where joints are nodes and bone connections are edges. However, existing GCN approaches typically use a fixed, manually defined graph structure that remains the same across all layers and all input samples. This is problematic because:

  1. Lack of Flexibility: The optimal graph structure for recognizing an action might vary depending on the specific action class (e.g., "wiping face" might require stronger hand-head connections than "jumping"). A fixed graph cannot adapt to these nuances.
  2. Hierarchical Gaps: GCNs are hierarchical, with deeper layers capturing more abstract features. A fixed graph topology might not be suitable for all levels of abstraction within the network.
  3. Limited Information: Most methods only consider the 2D/3D coordinates of joints (first-order information). They largely ignore the lengths and directions of bones (second-order information), which are highly discriminative for actions.

Methods

The paper proposes a novel Two-Stream Adaptive Graph Convolutional Network (2s-AGCN) that tackles these issues:

  • Adaptive Graph Construction: Instead of a fixed graph, the authors introduce an adaptive graph convolutional layer. This layer learns the graph topology dynamically. It does this by parameterizing three types of adjacency matrices:
  • A (Ak): The original, fixed physical structure of the human body.
  • B (Bk): A learnable adjacency matrix, optimized end-to-end with the network parameters. This allows the model to learn general, data-dependent graph structures.
  • C (Ck): A sample-specific graph, learned using a normalized Gaussian similarity function between joints. This provides maximum flexibility, adapting the graph to each individual input sample. These three matrices are combined, allowing the model to leverage both fixed anatomical constraints and data-driven, sample-specific relationships.

  • Two-Stream Framework: To incorporate second-order information, the model employs a two-stream architecture:

  • J-Stream: Processes the standard joint coordinate data (first-order information).
  • B-Stream: Processes bone information. Bones are represented as vectors pointing from a source joint to a target joint, capturing both length and direction. This stream is designed to be structurally similar to the J-stream for easier integration. The outputs of these two streams are fused (by summing their softmax scores) to produce the final prediction.

Datasets

The proposed model is evaluated on two large-scale, widely used datasets for skeleton-based action recognition:

  • NTU-RGBD [27]: Contains 56,000 action clips across 60 classes, captured by Kinect depth sensors. The paper uses the standard "Cross-Subject" (X-Sub) and "Cross-View" (X-View) benchmarks.
  • Kinetics-Skeleton [12]: A larger dataset with 300,000 video clips (though skeleton data is estimated from videos). The paper uses a subset of 400 classes.

Key Insights and Ideas

  • Data-Driven Graphs are Crucial: The experiments strongly suggest that fixed graph structures are suboptimal. Learning adaptive graphs, especially sample-specific ones, significantly improves performance.
  • Second-Order Information Matters: Explicitly modeling bone lengths and directions (second-order information) in a separate stream provides a substantial performance boost, highlighting its discriminative power.
  • Hierarchical Graph Adaptation: Visualizations show that different layers learn different graph topologies, with lower layers focusing on local connections and higher layers capturing more global, action-relevant relationships between distant joints.
  • Residual Connections Aid Stability: The use of residual connections in the adaptive graph convolutional blocks helps maintain stability and allows for easier integration into existing architectures.

Key Results

The 2s-AGCN achieves state-of-the-art performance on both the NTU-RGBD and Kinetics-Skeleton datasets.

  • On the NTU-RGBD X-View benchmark, the model achieves 95.1% accuracy, outperforming the previous state-of-the-art ST-GCN (88.3%) by a significant margin.
  • Ablation studies demonstrate the importance of each component:
  • The adaptive graph (A+B+C) yields 93.7% accuracy, while ST-GCN (fixed graph) achieves 92.7% (after re-optimization).
  • Removing any of the adaptive graph components (A, B, or C) leads to a drop in accuracy.
  • The two-stream approach (2s-AGCN) outperforms single-stream J-AGCN (93.7%) and B-AGCN (93.2%) on NTU-RGBD.

Limitations and Future Work

  • Computational Cost: While adaptive graphs improve accuracy, they likely increase computational complexity compared to fixed graphs. The paper doesn't explicitly detail the computational overhead.
  • Initialization: The paper mentions initializing the learnable graph components (Bk and embedding functions for Ck) to zero. Further exploration into optimal initialization strategies could be beneficial.
  • Future Work: The authors suggest that the code will be released to facilitate future research and communication, implying potential for further development and application of their adaptive graph approach. The inherent flexibility of the adaptive graph structure could be explored for other graph-based learning tasks beyond action recognition.

Summary updated

Abstract

In skeleton-based action recognition, graph convolutional networks (GCNs), which model the human body skeletons as spatiotemporal graphs, have achieved remarkable performance. However, in existing GCN-based meth- ods, the topology ofthe graph is set manually, and it is fixed over all layers and input samples. This may not be optimal for the hierarchical GCN and diverse samples in action recognition tasks. In addition, the second-order information (the lengths and directions of bones) of the skeleton data, which is naturally more informative and discriminative for action recognition, is rarely investigated in existing methods. In this work, we propose a novel two-stream adaptive graph convolutional network (2s-AGCN) for skeleton based action recognition. The topology of the graph in our model can be either uniformly or individually learned by the BP algorithm in an end-to-end manner. This data-driven method increases the flexibility of the model for graph construction and brings more generality to adapt to various data samples. Moreover, a two-stream framework is proposed to model both the first-order and the second-order information simultaneously, which shows notable improvement for the recognition accuracy. Extensive experiments on the two large-scale datasets, NTU-RGBD and Kinetics- Skeleton, demonstrate that the performance of our model exceeds the state-of-the-art with a significant margin.
BibTeX
@misc{shi2025two,
  author = {Lei Shi and Yifan Zhang and Jian Cheng and Hanqing Lu},
  title = {{Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition}},
  year = {2025},
  url = {https://openreview.net/forum?id=a6qzLgD2UP},
  howpublished = {OpenReview},
}
Figure