Multi Scale Temporal Graph Networks For Skeleton-based Action Recognition

Tingwei Li, Ruiwen Zhang, Qing Li

Abstract

Graph convolutional networks (GCNs) effectively capture features of related nodes, making them a popular choice for skeleton-based action recognition. However, existing GCN methods face two primary challenges: the neglect of spatiotemporal consistency due to separate spatial and temporal processing, and a reliance on physical joint connections that limits the learning of long-range relationships. To address these, we propose Temporal Graph Networks (TGN), which extract spatiotemporal features simultaneously, and a multi-scale graph strategy (full-scale, part-scale, and core-scale) to capture both local and global body contours. Our model, MS-TGN, achieves state-of-the-art performance on the NTU RGB+D and Kinetics-Skeleton datasets while maintaining high computational efficiency.

1. Introduction: The Challenges of Current GCNs

Skeleton-based action recognition is a robust alternative to video-based methods, as it focuses on joint movements while ignoring background noise. While Graph Convolutional Networks (GCNs) have become the standard for this task, they typically suffer from a "spatiotemporal split." Most models extract spatial features frame-by-frame using a spatial GCN and then pass these features to a separate Temporal Convolutional Network (TCN). This sequential approach fails to capture the inherent consistency between a joint's spatial position and its temporal trajectory.

Traditional Spatial GCN architecture which processes nodes frame-by-frame.

Furthermore, standard skeleton graphs are constrained by physical anatomy (e.g., connecting only adjacent joints like the elbow and wrist). This makes it difficult for models to recognize coordinated movements between distant body parts, such as the hands and feet, without passing information through many intermediate nodes.

2. Methodology

We introduce two key innovations to overcome these limitations: the Temporal Graph Network (TGN) and a Multi-Scale Graph Strategy.

2.1 Temporal Graph Networks (TGN)

The TGN redefines the fundamental unit of the graph. Instead of a node representing a single joint at a single point in time, a node in TGN represents a joint across the entire temporal sequence. This allows the network to process the spatial relationships between joints and their temporal evolution simultaneously within a single GCN block.

The proposed TGN block, which extracts spatiotemporal features simultaneously by treating joint sequences as nodes.

By eliminating the need for alternating spatial and temporal layers, the TGN preserves the consistency of movement data and simplifies the model architecture.

2.2 Multi-Scale Graph Strategy

To capture relationships beyond physical connections, we employ a hierarchical graph strategy:

  • Full-scale: Uses all 25 joints to capture fine-grained local movements.
  • Part-scale: Simplifies the skeleton to 11 key joints to capture the mid-level flow of body parts.
  • Core-scale: Focuses on 7 essential joints (torso, head, etc.) to capture global information and provide "shortcuts" between distant limbs.

This multi-scale approach increases the model's receptive field, allowing it to understand the global context of an action more effectively.

3. Experiments and Results

We evaluated MS-TGN on two major benchmarks: NTU RGB+D (60 action classes) and Kinetics-Skeleton (400 action classes).

3.1 Performance and Efficiency

MS-TGN demonstrated superior performance compared to existing state-of-the-art (SOTA) models:

  • Accuracy: On the NTU RGB+D (Cross-View) test, it achieved 95.9% accuracy. On Kinetics-Skeleton, it reached a Top-1 accuracy of 37.3%.
  • Efficiency: Despite its high accuracy, MS-TGN is remarkably efficient. It uses only 3.0 million parameters and 15.0 GFLOPs, making it approximately twice as efficient as Neural Architecture Search (NAS) models that it outperforms.

4. Discussion and Conclusion

The success of MS-TGN highlights that simultaneous spatiotemporal feature extraction is more effective than sequential processing. By improving the structural representation of the skeleton through multi-scale graphs, we achieve better results with a more efficient architecture.

Future work will focus on automatically learning optimal graph scales and extending the framework to handle noisier 2D skeleton data from monocular videos. MS-TGN provides a powerful and efficient foundation for real-time action recognition in various applications, from human-computer interaction to security monitoring.

Summary updated

Abstract

Graph convolutional networks (GCNs) can effectively capture the features of related nodes and improve the performance of the model. More attention is paid to employing GCN in Skeleton-Based action recognition. But existing methods based on GCNs have two problems. First, the consistency of temporal and spatial features is ignored for extracting features node by node and frame by frame. To obtain spatiotemporal features simultaneously, we design a generic representation of skeleton sequences for action recognition and propose a novel model called Temporal Graph Networks (TGN). Secondly, the adjacency matrix of the graph describing the relation of joints is mostly dependent on the physical connection between joints. To appropriately describe the relations between joints in the skeleton graph, we propose a multi-scale graph strategy, adopting a full-scale graph, part-scale graph, and core-scale graph to capture the local features of each joint and the contour features of important joints. Experiments were carried out on two large datasets and results show that TGN with our graph strategy outperforms state-of-the-art methods.
BibTeX
@misc{li2020multi,
  author = {Tingwei Li and Ruiwen Zhang and Qing Li},
  title = {{Multi Scale Temporal Graph Networks For Skeleton-based Action Recognition}},
  year = {2020},
  eprint = {2012.02970},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.02970},
}
Figure