GAS-GCN: Gated Action-Specific Graph Convolutional Networks for Skeleton-Based Action Recognition

Wensong Chan, Zhiqiang Tian, Yang WuSensors 2020 · OpenReview

TL;DR

This paper introduces GAS-GCN, a novel approach for skeleton-based action recognition that dynamically learns action-specific relationships between body joints, going beyond fixed structural connections. It also incorporates a gated temporal convolution to effectively filter temporal information, leading to state-of-the-art performance on several benchmark datasets.

What Problem Does This Paper Address?

Recognizing human actions from skeletal data has become a crucial task in areas like human-computer interaction and intelligent monitoring. While Graph Convolutional Networks (GCNs) have shown great promise, existing methods often rely on pre-defined, static "structural" connections between body joints. This overlooks "implicit" connections that are crucial for distinguishing subtle actions and can vary significantly between different activities. For instance, the relationship between hands is vital for recognizing "rubbing hands" but less so for "nodding." Furthermore, effectively processing the temporal sequence of these skeletal poses, while filtering out irrelevant information, remains a challenge.

Methods

The core contributions of GAS-GCN lie in two novel components:

  1. Action-Specific Graph Convolutional Module (ASGCM): This module addresses the limitations of fixed structural edges. It learns to infer "implicit" connections between joints and dynamically determines the optimal ratio of structural to implicit edges for each specific action. This is achieved by using the input skeleton data to generate an action-specific adjacency matrix, allowing the GCN to adapt its spatial feature extraction to the nuances of each action.
  2. Gated Temporal Convolution: To handle the temporal dimension more effectively, the authors propose a gated convolutional network. This mechanism acts like a filter, allowing the network to selectively retain important temporal information relevant to action recognition while discarding redundant or noisy data. This is inspired by gating mechanisms in recurrent neural networks but implemented with convolutional layers for computational efficiency.

These two components are integrated within a cascaded, residual framework to enhance performance and mitigate gradient vanishing issues. The architecture also incorporates channel-wise attention (SE block) for further feature refinement.

Datasets

The proposed GAS-GCN model was evaluated on three large-scale datasets:

  • NTU-RGB + D [16]: A widely used dataset for skeleton-based action recognition, containing 3D joint data for 60 action classes.
  • NTU-RGB + D 120 [17]: An extension of NTU-RGB + D, featuring a larger number of action classes (120) and videos.
  • Kinetics [18]: A significantly larger dataset, though the original data is RGB-based. The authors utilized extracted skeleton data for evaluation.

Key Insights and Ideas

  • Implicit Connections are Key: The paper strongly emphasizes that static, hand-crafted skeletal connections are insufficient. Learning action-specific implicit connections is crucial for fine-grained action recognition.
  • Dynamic Edge Weighting: The ability to dynamically adjust the importance of structural vs. implicit edges based on the action is a significant advancement. This allows the model to be more sensitive to subtle, action-dependent joint relationships.
  • Gated Temporal Filtering: Simply applying temporal convolutions can lead to information loss or the propagation of irrelevant features. Gated convolutions offer a more controlled and effective way to process temporal sequences.
  • Two-Stream Fusion Benefits: Combining joint and bone information in a two-stream architecture further boosts performance, indicating that different representations of skeletal data are complementary.

Key Results

GAS-GCN achieved state-of-the-art results on the NTU-RGB + D and Kinetics datasets. Specifically:

  • On NTU-RGB + D, GAS-GCN achieved 90.4% accuracy in cross-subject evaluation and 96.5% in cross-view evaluation, outperforming previous methods like 2s-AGCN and AGC-LSTM.
  • On Kinetics, it reached 37.8% top-1 accuracy and 60.9% top-5 accuracy, also setting a new benchmark.
  • Ablation studies confirmed the effectiveness of both the Action-Specific Graph Convolutional Module (ASGCM) and the Gated Temporal Convolution, with their combination yielding the best performance.
  • The two-stream fusion approach further improved results, reaching 96.5% on NTU-RGB + D (cross-view).

Limitations and Future Work

  • Computational Cost: While the gated CNN is more efficient than LSTM, the paper notes a ~20% increase in GPU memory and time consumption compared to the baseline AGCN, suggesting potential for further optimization.
  • Generalizability: The authors suggest that the action-specific graph convolution and gated convolution mechanisms could be applied to other computer vision tasks beyond action recognition. This hints at future work exploring their broader applicability.
  • Dataset Limitations: The NTU-RGB + D 120 dataset was relatively new at the time of publication, limiting direct comparisons with other recent methods. The paper reports strong results on this dataset, but further validation with more advanced methods would be beneficial.

Summary updated

Abstract

Skeleton-based action recognition has achieved great advances with the development of graph convolutional networks (GCNs). Many existing GCNs-based models only use the fixed hand-crafted adjacency matrix to describe the connections between human body joints. This omits the important implicit connections between joints, which contain discriminative information for different actions. In this paper, we propose an action-specific graph convolutional module, which is able to extract the implicit connections and properly balance them for each action. In addition, to filter out the useless and redundant information in the temporal dimension, we propose a simple yet effective operation named gated temporal convolution. These two major novelties ensure the superiority of our proposed method, as demonstrated on three large-scale public datasets: NTU-RGB + D, Kinetics, and NTU-RGB + D 120, and also shown in the detailed ablation studies.
BibTeX
@inproceedings{chan2020gas,
  author = {Wensong Chan and Zhiqiang Tian and Yang Wu},
  title = {{GAS-GCN: Gated Action-Specific Graph Convolutional Networks for Skeleton-Based Action Recognition}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=uc_IUi7mX2},
}
Figure