Abstract
Graph convolutional networks (GCNs) effectively capture features of related nodes, making them a popular choice for skeleton-based action recognition. However, existing GCN methods face two primary challenges: the neglect of spatiotemporal consistency due to separate spatial and temporal processing, and a reliance on physical joint connections that limits the learning of long-range relationships. To address these, we propose Temporal Graph Networks (TGN), which extract spatiotemporal features simultaneously, and a multi-scale graph strategy (full-scale, part-scale, and core-scale) to capture both local and global body contours. Our model, MS-TGN, achieves state-of-the-art performance on the NTU RGB+D and Kinetics-Skeleton datasets while maintaining high computational efficiency.
1. Introduction: The Challenges of Current GCNs
Skeleton-based action recognition is a robust alternative to video-based methods, as it focuses on joint movements while ignoring background noise. While Graph Convolutional Networks (GCNs) have become the standard for this task, they typically suffer from a "spatiotemporal split." Most models extract spatial features frame-by-frame using a spatial GCN and then pass these features to a separate Temporal Convolutional Network (TCN). This sequential approach fails to capture the inherent consistency between a joint's spatial position and its temporal trajectory.
Furthermore, standard skeleton graphs are constrained by physical anatomy (e.g., connecting only adjacent joints like the elbow and wrist). This makes it difficult for models to recognize coordinated movements between distant body parts, such as the hands and feet, without passing information through many intermediate nodes.
2. Methodology
We introduce two key innovations to overcome these limitations: the Temporal Graph Network (TGN) and a Multi-Scale Graph Strategy.
2.1 Temporal Graph Networks (TGN)
The TGN redefines the fundamental unit of the graph. Instead of a node representing a single joint at a single point in time, a node in TGN represents a joint across the entire temporal sequence. This allows the network to process the spatial relationships between joints and their temporal evolution simultaneously within a single GCN block.
By eliminating the need for alternating spatial and temporal layers, the TGN preserves the consistency of movement data and simplifies the model architecture.
2.2 Multi-Scale Graph Strategy
To capture relationships beyond physical connections, we employ a hierarchical graph strategy:
- Full-scale: Uses all 25 joints to capture fine-grained local movements.
- Part-scale: Simplifies the skeleton to 11 key joints to capture the mid-level flow of body parts.
- Core-scale: Focuses on 7 essential joints (torso, head, etc.) to capture global information and provide "shortcuts" between distant limbs.
This multi-scale approach increases the model's receptive field, allowing it to understand the global context of an action more effectively.
3. Experiments and Results
We evaluated MS-TGN on two major benchmarks: NTU RGB+D (60 action classes) and Kinetics-Skeleton (400 action classes).
3.1 Performance and Efficiency
MS-TGN demonstrated superior performance compared to existing state-of-the-art (SOTA) models:
- Accuracy: On the NTU RGB+D (Cross-View) test, it achieved 95.9% accuracy. On Kinetics-Skeleton, it reached a Top-1 accuracy of 37.3%.
- Efficiency: Despite its high accuracy, MS-TGN is remarkably efficient. It uses only 3.0 million parameters and 15.0 GFLOPs, making it approximately twice as efficient as Neural Architecture Search (NAS) models that it outperforms.
4. Discussion and Conclusion
The success of MS-TGN highlights that simultaneous spatiotemporal feature extraction is more effective than sequential processing. By improving the structural representation of the skeleton through multi-scale graphs, we achieve better results with a more efficient architecture.
Future work will focus on automatically learning optimal graph scales and extending the framework to handle noisier 2D skeleton data from monocular videos. MS-TGN provides a powerful and efficient foundation for real-time action recognition in various applications, from human-computer interaction to security monitoring.
Summary updated