Spatio-Temporal Difference Descriptor for Skeleton-Based Action Recognition

Chongyang Ding, Kai Liu, Jari Korhonen, Evgeny BelyaevAAAI 2021 · DOI

Abstract

Skeleton-based action recognition is a critical task in computer vision, offering robustness against environmental variations. However, existing Graph Convolutional Networks (GCNs) often fail to capture the subtle intra-frame joint differences and inter-frame motion dynamics essential for distinguishing complex actions. This paper introduces a novel framework that combines a directional convolution operation with a spatio-temporal difference descriptor based on Symmetric Positive Definite (SPD) matrices. By utilizing rotation matrices to preserve geometric scale and learning on Riemannian manifolds, the proposed method achieves state-of-the-art results across multiple benchmarks, including a 3% accuracy boost on the G3D-Gaming dataset.

Introduction

Human action recognition from skeleton data (3D joint positions) has gained popularity due to its resilience to lighting and background changes. While standard GCNs are effective at modeling joint dependencies, they possess a significant "blind spot": they struggle to identify intra-frame differences (static pose signatures) and inter-frame dynamics (joint displacements over time). Without these features, models often fail to distinguish between actions with similar joint configurations but different movement patterns.

Illustration of intra-frame differences and inter-frame dynamics captured by the proposed descriptors.

Methodology

The proposed architecture, Di-StddNet, introduces three primary innovations to address these limitations:

1. Directional Convolution

Unlike standard convolutions that use scalar weights, this method partitions the 3D space around a joint into $b$ regions. It assigns unique 3D rotation matrices as weights based on the direction of a neighboring joint. This approach allows the network to learn spatial dependencies while preserving the physical scale and geometric orientation of the skeleton.

2. Spatio-Temporal Difference Descriptor

The model generates two types of statistical descriptors represented as Symmetric Positive Definite (SPD) matrices:

  • Spatial Difference Matrix: Captures the unique static pattern of a skeleton by analyzing the variance between all joints within a single frame.
  • Temporal Dynamic Matrix: Captures motion by calculating joint displacements between adjacent frames.

3. Deep SPD Metric Learning

Since SPD matrices reside on a curved Riemannian manifold, standard Euclidean layers are unsuitable. The authors developed a 2D fully connected layer for dimensionality reduction and a novel rectification function. This function prevents eigenvalues from becoming too small, ensuring numerical stability during the "log-map" operation required to flatten the manifold for final classification.

The complete Di-StddNet architecture, showing the transition from skeleton sequences to SPD metric learning.

Experimental Results

The Di-StddNet was evaluated on three standard benchmarks, consistently outperforming existing GCN and manifold-based methods:

  • G3D-Gaming: Achieved 92.03% accuracy, a ~3% improvement over the previous state-of-the-art (LieNet).
  • HDM05: Reached 82.32%, surpassing the strong ST-GCN baseline.
  • Northwestern-UCLA: Achieved 93.8%, outperforming the AGC-LSTM model.

Ablation studies confirmed that both the directional convolution and the non-linear SPD mapping blocks were essential for these performance gains.

Conclusion and Future Work

By focusing on the differences between joints rather than just their absolute positions, and by respecting the geometric nature of the human body through rotation matrices, this framework sets a new standard for skeleton-based recognition. Future research may address the computational complexity of SPD manifold operations and extend the temporal descriptors to capture long-range dynamics beyond adjacent frames.

Summary updated

BibTeX
@inproceedings{ding2021spatio,
  author = {Chongyang Ding and Kai Liu and Jari Korhonen and Evgeny Belyaev},
  title = {{Spatio-Temporal Difference Descriptor for Skeleton-Based Action Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i2.16210},
}
Figure