Spatio-Temporal Fusion for Human Action Recognition via Joint Trajectory Graph

Yaolin Zheng, Hongbo Huang, Xiuying Wang, Xiaoxu Yan, Longfei XuAAAI 2024 · DOI

TL;DR

This paper introduces the Joint Trajectory Graph (JTG) and JT-GraphFormer model for human action recognition from skeleton data. By explicitly modeling spatio-temporal relationships between joint trajectories, the model achieves state-of-the-art performance on benchmark datasets, addressing limitations of existing GCN and Transformer approaches.

What Problem Does This Paper Address?

Human action recognition from skeleton data is a crucial task in computer vision with applications in human-computer interaction and intelligent surveillance. While Graph Convolutional Networks (GCNs) excel at capturing spatial relationships and Transformers are good at long-range dependencies, existing methods often struggle to effectively integrate both spatial and temporal information into a unified representation. GCNs typically rely on static spatial graphs, limiting their ability to capture dynamic temporal evolution. Transformers, on the other hand, may lack explicit topological structure information, impacting robustness. This paper aims to bridge this gap by proposing a novel graph representation and a Transformer-based architecture that explicitly models spatio-temporal dependencies within joint trajectories.

Methods

The core contribution is the Joint Trajectory Graph (JTG), which extends the traditional spatial graph of human joints by incorporating a temporal dimension. This creates a richer graph structure that captures the dynamic evolution of joint positions over a sequence of frames.

The proposed model, JT-GraphFormer, leverages this JTG. Key components include:

  • Spatio-Temporal Dijkstra Attention (STDA): This mechanism enhances the self-attention in Transformers by incorporating shortest path distances between joints within the JTG. This allows the model to prioritize attention to more relevant neighboring joints, improving the capture of spatio-temporal relationships.
  • TCN Module: A temporal convolutional network module is used for sequential aggregation of joint trajectory features.
  • Koopman Operator: For the classification stage, the Koopman operator is introduced. This linear operator is used to model the evolution of extracted features in an infinite-dimensional Hilbert space, effectively linearizing non-linear dynamical systems and improving classification performance and robustness.

Datasets

The paper evaluates the proposed JT-GraphFormer on three widely used skeleton-based action recognition datasets:

  • NTU RGB+D (NTU-60): Contains 56,880 skeleton action sequences across 60 classes.
  • NTU RGB+D 120 (NTU-120): An extension of NTU-60 with 114,480 samples and 120 classes.
  • Northwestern-UCLA (N-UCLA): Features 1,494 video clips in 10 classes.

The paper uses standard cross-subject (XSub) and cross-view (XView) evaluation protocols for these datasets.

Key Insights and Ideas

  • Explicit Spatio-Temporal Topology is Crucial: The paper strongly advocates for explicitly modeling the dynamic, spatio-temporal relationships of joint trajectories, rather than relying solely on static spatial connections or purely temporal sequence modeling.
  • Dijkstra for Attention Guidance: Incorporating shortest path distances (via Dijkstra) into the attention mechanism provides a principled way to guide the model's focus on relevant spatio-temporal information, leading to better feature aggregation.
  • Koopman Operator for Robust Classification: Using the Koopman operator for classification offers a novel way to handle the inherent non-linear dynamics of human actions by linearizing feature evolution, leading to improved generalization.
  • Dynamic JTG Outperforms Static: The experiments show that a dynamic JTG, which adapts its structure over time, is more effective than a static one, highlighting the importance of capturing temporal changes in relationships.

Key Results

The JT-GraphFormer model achieves state-of-the-art performance on all three benchmark datasets:

  • NTU-60 (X-Sub): Achieved 93.4% Top-1 accuracy with a four-stream ensemble.
  • NTU-120 (X-Sub): Achieved 89.9% Top-1 accuracy with a four-stream ensemble.
  • N-UCLA: Achieved 97.2% Top-1 accuracy with a four-stream ensemble.

The paper also demonstrates that even the "joint only" modality of JT-GraphFormer outperforms many existing methods, showcasing the effectiveness of the core JTG and STDA mechanisms. Ablation studies confirm the benefits of dynamic JTG, inter-layer weight sharing, and the Koopman operator.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that testing on larger datasets like Kinetics-400 might be challenging due to the model's parameter count, especially with the Koopman operator.
  • Limited Stream Fusion: The current model considers integrating up to four streams. Future work could explore fusing more modalities to further exploit potential.
  • Structured Data Focus: The current framework is designed for structured data sequences like human motion. Extending it to other types of structured data could be a future direction.
  • Unsupervised Learning: The paper suggests that exploring unsupervised or semi-supervised learning approaches for JT-GraphFormer is a promising area for future research.

Summary updated

BibTeX
@inproceedings{zheng2024spatio,
  author = {Yaolin Zheng and Hongbo Huang and Xiuying Wang and Xiaoxu Yan and Longfei Xu},
  title = {{Spatio-Temporal Fusion for Human Action Recognition via Joint Trajectory Graph}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28590},
}
Figure