Efficient Skeleton-Based Action Recognition via Joint-Mapping Strategies

Min-Seok Kang, Dongoh Kang, HanSaem KimWACV 2023

TL;DR

This paper tackles the challenge of making skeleton-based action recognition models lightweight and efficient for real-world deployment. They introduce novel "joint-mapping" modules that reduce the number of joints processed in intermediate layers of Graph Convolutional Networks (GCNs), leading to significant reductions in computational cost and latency without sacrificing accuracy.

What Problem Does This Paper Address?

Skeleton-based action recognition has shown great promise for surveillance and other applications because it's less sensitive to background noise and requires less data than video-based methods. However, current state-of-the-art models, often based on Graph Convolutional Networks (GCNs), are computationally expensive and have large model sizes, making them unsuitable for real-time, embedded systems. This is particularly problematic for applications like automated surveillance where immediate, reliable decisions are crucial, and where models need to run on resource-constrained hardware. The authors highlight that while GCNs are powerful, they often maintain the full set of joints throughout all layers, leading to unnecessary computational burden.

Methods

The core contribution of this paper lies in their proposed joint-mapping modules, designed to reduce the number of joints processed in the middle layers of GCNs. This is inspired by the hierarchical structure of Convolutional Neural Networks (CNNs) that progressively reduce spatial dimensions. They propose two strategies:

  • Manual Mapping (MM): This approach leverages prior knowledge of human body structure. The J input joints are manually divided into five parts: left arm, right arm, left leg, right leg, and the main body. These parts are then processed by specialized "encoders" (using a bottleneck fusion and cross-attention mechanism) to produce a reduced set of J' (specifically 5) joint representations. This method aims to learn discriminative parts and remove redundant dependencies.
  • Adaptive Mapping (AM): This method learns to map the J joints to a reduced set of J' nodes without relying on prior human knowledge. It uses a trainable mapping matrix (MG) that learns the optimal way to represent the joints in a lower-dimensional space. This approach is more flexible and can discover non-intuitive mappings that might be beneficial for action classification.

These joint-mapping modules are integrated into a multi-branch GCN architecture (similar to CTR-GCN) that also utilizes bone and velocity information as additional input modalities. The overall pipeline involves initial GCN and temporal convolution layers for joint embedding, followed by the joint-mapping module, and finally, a classification layer.

Datasets

The paper evaluates their proposed methods on two widely used large-scale datasets for skeleton-based action recognition:

  • NTU RGB+D (NTU60): This dataset contains 56,880 video clips with 60 action classes, captured by three Kinect V2 cameras. It's evaluated on two benchmarks: cross-subject (X-sub) and cross-view (X-view).
  • NTU RGB+D 120 (NTU120): This is an extended version of NTU60, featuring 114,480 video clips and 120 action classes. It's evaluated on cross-subject (X-sub) and cross-setup (X-set) benchmarks.

Key Insights and Ideas

  • Reducing Joint Representation is Key for Efficiency: The fundamental insight is that not all joints are equally important at every stage of processing. By strategically reducing the number of joints in intermediate layers, significant computational savings can be achieved without a proportional loss in accuracy.
  • Manual vs. Adaptive Mapping Trade-offs: The paper demonstrates that both manually defined (MM) and learned (AM) joint mappings can be effective. MM offers interpretability and leverages domain knowledge, while AM provides greater flexibility and can potentially discover more optimal representations.
  • Latency is Crucial for Real-World Deployment: The authors emphasize that latency, not just FLOPs or parameter count, is a critical metric for practical applications. Their use of TensorRT for latency measurement highlights this focus.
  • Hierarchical Structure Benefits GCNs: The inspiration from CNNs' hierarchical feature reduction suggests that similar principles can be applied to GCNs to improve their efficiency.

Key Results

The proposed MM-GCN and AM-GCN models achieve a strong accuracy-latency trade-off.

  • On NTU60 (X-sub), MM-GCN achieves 90.8% accuracy with 13.2 ms latency, while AM-GCN achieves 90.3% accuracy with 12.5 ms latency.
  • These results are competitive with or superior to state-of-the-art methods, especially when considering latency. For instance, their models are significantly faster than CTR-GCN (14.1 ms) while maintaining comparable or better accuracy.
  • The paper also shows that their models can achieve good performance with fewer input modalities (e.g., just joints), further reducing computational load.
  • When deployed on mobile devices (iPhone XR and Jetson AGX Xavier), their models demonstrate substantial latency reductions compared to CTR-GCN, making them suitable for edge computing.

Limitations and Future Work

  • Manual Mapping Limitations: The manual mapping relies on prior knowledge, which might not always be optimal for all action classes or datasets. The paper acknowledges that the pie charts for adaptive mapping are more complex, suggesting that learned mappings can be less intuitive.
  • Future Work: The authors suggest extending their work to future research on skeleton-based action recognition and applying it to real-world applications. They also mention that the position of the joint-mapping modules can be further explored, although they found layer 6 to be optimal in their experiments. The paper also hints at future research in exploring more sophisticated joint-mapping strategies and their impact on interpretability.

Summary updated

BibTeX
@inproceedings{kang2023efficient,
  author = {Min-Seok Kang and Dongoh Kang and HanSaem Kim},
  title = {{Efficient Skeleton-Based Action Recognition via Joint-Mapping Strategies}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure