Skeleton Based Human Action Recognition with Global Context-Aware Attention LSTM Networks

Jun Liu, Gang Wang, Ling-Yu Duan, Ping Hu, Alex C. KotIEEE TIP 2017 · DOI · OpenReview

TL;DR

This paper introduces the Global Context-Aware Attention LSTM (GCA-LSTM) network for 3D skeleton-based human action recognition. By incorporating a global context memory cell and a recurrent attention mechanism, GCA-LSTM selectively focuses on informative skeletal joints, significantly improving recognition accuracy. A novel two-stream architecture further enhances performance by combining fine-grained (joint-level) and coarse-grained (body part-level) attention.

What Problem Does This Paper Address?

Recognizing human actions from 3D skeleton sequences is crucial for applications like surveillance and human-computer interaction. While Long Short-Term Memory (LSTM) networks are adept at modeling sequential data, they struggle with identifying which parts of the skeleton are most relevant for a given action. Irrelevant joint movements can introduce noise and degrade performance. Existing LSTMs lack an explicit mechanism to "pay attention" to the most informative joints, especially when considering the global context of the entire action sequence, which is vital for accurate classification.

Methods

The core innovation is the Global Context-Aware Attention LSTM (GCA-LSTM). This architecture extends the standard LSTM by introducing a global context memory cell. This cell stores a representation of the entire action sequence, allowing the network to assess the informativeness of each joint at every time step.

Key components include:

  • Global Context Memory Cell: This cell is initialized using the outputs of an initial LSTM layer and is iteratively refined. It provides a global view of the action.
  • Recurrent Attention Mechanism: This allows the network to progressively improve its attention capabilities. After an initial attention pass, the refined global context memory is used to perform attention again, leading to a more accurate focus on informative joints. This process can be repeated for multiple "attention iterations."
  • Informativeness Gate: The attention mechanism generates an "informativeness score" for each joint, which acts as a gate within the LSTM unit. This gate controls how much information from the current joint contributes to updating the LSTM's cell state, effectively filtering out irrelevant data.
  • Two-Stream GCA-LSTM: To further boost performance, the authors propose a two-stream architecture. One stream handles fine-grained attention (focusing on individual joints), while the other handles coarse-grained attention (focusing on entire body parts). These streams operate in parallel, each with its own global context memory, and their predictions are combined.

Datasets

The proposed methods were evaluated on the following datasets:

  • NTU RGB+D [55]: A large-scale dataset with over 56,000 video samples and 60 action classes, collected with Kinect V2.
  • SYSU-3D [71]: Contains 480 skeleton sequences across 12 action classes, also collected with Kinect.
  • UT-Kinect [47]: Features noisy skeleton sequences from a stationary Kinect, with 10 action classes.
  • SBU-Kinect Interaction [72]: Focuses on two-person interaction recognition with 8 action classes.
  • Berkeley MHAD [73]: Collected using a motion capture system, providing more accurate skeleton data across 11 action classes.

Key Insights and Ideas

  • Global Context is Crucial for Attention: The paper strongly argues that to effectively identify informative joints, the network needs to understand the global context of the entire action sequence, not just the local context of the previous time step.
  • Recurrent Attention Improves Focus: Iteratively refining the global context memory through multiple attention passes significantly enhances the network's ability to pinpoint relevant joints and body parts.
  • Body Part-Level Attention Complements Joint-Level: Combining fine-grained (joint) and coarse-grained (body part) attention provides a more comprehensive understanding of actions, especially those involving coordinated movements of entire limbs.
  • Stepwise Training Enhances Stability and Efficiency: A proposed stepwise training scheme, where parameters are optimized incrementally across attention iterations, leads to faster convergence and better generalization compared to direct end-to-end training.
  • Robustness to Noise: The GCA-LSTM demonstrates remarkable robustness to noisy skeleton data, a common issue with depth sensor inputs.

Key Results

The proposed GCA-LSTM network achieved state-of-the-art performance on all five evaluated datasets.

  • NTU RGB+D: Outperformed the original ST-LSTM [29] by 6.9% (CS) and 6.3% (CV). The two-stream GCA-LSTM with stepwise training achieved 77.1% (CS) and 85.1% (CV).
  • SYSU-3D: Achieved 78.6% accuracy with direct training and 79.1% with stepwise training for the two-stream GCA-LSTM.
  • UT-Kinect: Reached 99.0% accuracy with stepwise training.
  • Berkeley MHAD: Achieved 100% accuracy.
  • SBU-Kinect Interaction: Demonstrated the best performance among compared methods.
  • Robustness: Even with significant Gaussian noise (σ=12cm), the GCA-LSTM maintained 92.7% accuracy on the Berkeley MHAD dataset.

Limitations and Future Work

  • GPU Memory Limitations: The authors mention that they did not explore more than three attention iterations due to GPU memory constraints. Future work could investigate more efficient ways to handle higher numbers of iterations.
  • Parameter Sharing: While parameter sharing within iterations improved generalization, sharing across iterations was found to potentially limit representation capacity. The paper settled on sharing within iterations. Further exploration of optimal parameter sharing strategies could be beneficial.
  • Soft Attention vs. Hard Attention: The paper briefly mentions exploring soft attention for visualization, noting a slight drop in accuracy compared to their proposed gated mechanism. Further investigation into different attention mechanisms could be a future direction.
  • Computational Cost: While stepwise training improves efficiency, the overall GCA-LSTM architecture, especially with multiple attention iterations and two streams, is computationally intensive. Optimizing for speed and efficiency remains an ongoing challenge in action recognition.

Summary updated

Abstract

Human action recognition in 3D skeleton sequences has attracted a lot of research attention. Recently, Long Short-Term Memory (LSTM) networks have shown promising performance in this task due to their strengths in modeling the dependencies and dynamics in sequential data. As not all skeletal joints are informative for action recognition, and the irrelevant joints often bring noise which can degrade the performance, we need to pay more attention to the informative ones. However, the original LSTM network does not have explicit attention ability. In this paper, we propose a new class of LSTM network, Global Context-Aware Attention LSTM (GCA-LSTM), for skeleton based action recognition. This network is capable of selectively focusing on the informative joints in each frame of each skeleton sequence by using a global context memory cell. To further improve the attention capability of our network, we also introduce a recurrent attention mechanism, with which the attention performance of the network can be enhanced progressively. Moreover, we propose a stepwise training scheme in order to train our network effectively. Our approach achieves state-of-the-art performance on five challenging benchmark datasets for skeleton based action recognition.
BibTeX
@article{liu2017skeleton,
  author = {Jun Liu and Gang Wang and Ling-Yu Duan and Ping Hu and Alex C. Kot},
  title = {{Skeleton Based Human Action Recognition with Global Context-Aware Attention LSTM Networks}},
  journal = {IEEE Trans. Image Process.},
  year = {2017},
  doi = {10.1109/tip.2017.2785279},
}
Figure