A Lightweight Attentional Shift Graph Convolutional Network for Skeleton-Based Action Recognition

Xianshan Li, Jingwen Kang, Yang Yang, Fengda ZhaoInternational Journal of Computers Communications & Control 2023 · DOI

This paper, published in the International Journal of Computers Communications & Control, presents a new approach to making human action recognition more efficient for real-world deployment.

TL;DR

The authors introduce LA-SGCN, a lightweight Graph Convolutional Network that slashes computational costs and parameter counts by up to 6x compared to previous state-of-the-art models. By combining "shift" convolutions with an early-fusion multi-stream architecture, they achieve high-accuracy action recognition that is lean enough to run on mobile and embedded devices.

What Problem Does This Paper Address?

Skeleton-based action recognition uses 3D coordinates of human joints to identify activities like walking, waving, or falling. While Graph Convolutional Networks (GCNs) have become the gold standard for this task, they are notoriously "heavy." High-accuracy models often require massive amounts of memory and processing power (FLOPs), making them nearly impossible to run on mobile phones or smart home sensors.

The challenge is a classic trade-off: how do you reduce the "spatiotemporal complexity" (the math required to track joints over time) without losing the fine-grained details needed to tell the difference between, say, "brushing teeth" and "shaving"?

Methods

The authors propose three primary innovations to create the Lightweight Architecture Attentional Shift Graph Convolutional Network (LA-SGCN):

  1. Shift Graph Convolution: Instead of performing heavy matrix multiplications to aggregate data from neighboring joints, this method "shifts" feature channels in memory. This effectively expands the receptive field (what the model "sees") at a fraction of the computational cost.
  2. Spatial Shift: Treats the skeleton as a complete graph, allowing every node to share information.
  3. Temporal Shift: Uses a learnable parameter to adaptively decide how many frames of movement to look at.
  4. Shallow Multi-stream Early Fusion: Most models use four separate "streams" (Joints, Bones, and their velocities) and run them through four full networks before merging them at the very end. LA-SGCN merges these streams after just three layers, significantly reducing the total number of layers and parameters.
  5. Efficient Channel Attention (ECA): The model adds a lightweight attention mechanism that helps the network focus on the most important feature channels (e.g., the movement of the hands during a "clapping" action) without the overhead of traditional dimensionality reduction.

Datasets

The model was evaluated on three industry-standard benchmarks:

  • NTU RGB+D: A massive dataset with 56,880 samples and 60 action classes.
  • NTU-120 RGB+D: An expanded version of the above, featuring 120 action classes.
  • Northwestern-UCLA: A smaller but challenging dataset captured from multiple camera angles.

Key Insights and Ideas

  • Shifting is "Cheaper" than Multiplying: The core "aha moment" is that shifting data in memory can replace complex graph operations. By concentrating the heavy math into simple point-wise convolutions, the model remains expressive but computationally light.
  • The Diminishing Returns of Depth: The authors found that because shift convolutions provide such a broad receptive field, they didn't need the 10-layer depth used by previous models. Reducing the network to 8 layers and fusing streams early provided the best balance of speed and accuracy.
  • Learnable Receptive Fields: By making the temporal shift parameter learnable, the model doesn't just look at a fixed window of time; it "tunes" itself to the rhythm of the specific action it is trying to identify.

Key Results

The LA-SGCN model achieved impressive efficiency gains while remaining competitive on accuracy:

  • Parameter Reduction: On the NTU RGB+D dataset, LA-SGCN used only 0.46M parameters, compared to 2.76M for the previous 4s Shift-GCN—a 6x reduction.
  • Computational Speedup: The model required only 2.8G FLOPs, whereas the baseline required 10.0G (a 3.6x reduction).
  • Accuracy: Despite being much smaller, it maintained high performance, achieving 90.5% accuracy on the NTU X-sub task (only 0.2% lower than the much heavier baseline) and actually outperforming the baseline on the Northwestern-UCLA dataset (95.7% vs 94.6%).

Limitations and Future Work

The authors acknowledge that while the model is efficient, the process of finding the "optimal" architecture (the number of layers before and after fusion) was done using a "cumbersome" control variable method. They suggest that future versions could use automated neural architecture search to find even better configurations.

Additionally, the current model assumes "complete" skeleton data. In the real world, body parts are often obscured (occlusion). The authors plan to explore how this lightweight architecture handles incomplete or noisy skeleton data in future research.

Summary updated

Abstract

In the field of skeleton-based human behavior recognition, graph convolutional neural networks have made remarkable achievements. However, high precision networks are often accompanied by numerous parameters and computational cost, and their application in mobile devices has considerable limitations. Aiming at the problem of excessive spatiotemporal complexity of high-accuracy methods, this paper further analyzes the lightweight human action recognition model and proposes a lightweight architecture attentional shift graph convolutional network. There are three main improvements in this model. Firstly, shift convolution is a lightweight convolution method that can be combined with graph convolution to effectively reduce its complexity. At the same time, a shallow architecture for multi-stream early fusion is designed to reduce the network scale by merging multi-stream networks and reducing the number of network layers. In addition, the efficient channel attention module is introduced into the model to capture the underlying characteristic information in the channel domain. Experiments are conducted on the three existing skeleton datasets, NTU RGB+D, NTU-120 RGB+D, and Northwestern-UCLA. Results demonstrate that the proposed model is not only competitive in accuracy, but also outperforms current mainstream methods in parameter count and computational cost, and supports running in some devices with limited computing and storage resources.
BibTeX
@article{li2023lightweight,
  author = {Xianshan Li and Jingwen Kang and Yang Yang and Fengda Zhao},
  title = {{A Lightweight Attentional Shift Graph Convolutional Network for Skeleton-Based Action Recognition}},
  journal = {International Journal of Computers Communications \& Control},
  year = {2023},
  doi = {10.15837/ijccc.2023.3.5061},
}
Figure