Skeleton-Based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module.

Chenyang Li, Xin Zhang, Lufan Liao, Lianwen Jin, Weixin Yang

Abstract

This paper presents a highly efficient framework for skeleton-based gesture recognition that leverages Path Signature (PS) features and a novel Temporal Transformer Module (TTM). By shifting the complexity of temporal modeling to the feature extraction phase, the proposed method replaces resource-intensive Recurrent Neural Networks (RNNs) and 3D Convolutional Neural Networks (3D-CNNs) with simple Fully Connected (FC) layers. The framework achieves state-of-the-art performance on major benchmarks while requiring significantly fewer computational resources—demonstrating a 76x reduction in Multiply-Adds compared to traditional LSTM-based approaches.

Introduction

Recognizing hand gestures from 3D skeleton data is a complex task due to significant intra-class variations in speed, duration, and execution style. While deep learning models like LSTMs and 3D-CNNs are effective at capturing temporal dependencies, they are computationally expensive and often difficult to train on smaller datasets. Conversely, hand-crafted features often lack the representational power needed for complex dynamics. This research proposes a "middle ground" by using Path Signatures—robust mathematical descriptors of trajectories—to represent motion, allowing for a lightweight classification model based on FC layers.

Methodology

System Overview

The proposed framework consists of three main components: the Attention on Hand (AOH) principle, the Path Signature (PS) extractor, and the Temporal Transformer Module (TTM).

Overview of the proposed gesture recognition framework featuring AOH, TTM, and PS extraction.

Path Signature (PS) Features and AOH

Path Signatures are a collection of iterated integrals that provide a unique representation for paths. To manage the dimensionality of these features, the "Attention on Hand" (AOH) principle focuses on joints most relevant to gestures (elbows, wrists, fingers) and their relative positions to the rest of the body. This keeps the feature vector compact while focusing on the most discriminative parts of a gesture.

The framework extracts three types of PS features:

  1. Spatial PS (S_PS): Captures geometric relationships between joints within a single frame.
  2. Temporal PS (T_PS): Captures the evolution of individual joints over time.
  3. Temporal Spatial PS (T_S_PS): Captures how spatial correlations between joints evolve over time.
Detailed view of the AOH principle and the extraction of spatial and temporal Path Signature features.

Temporal Transformer Module (TTM)

To handle variations in gesture speed, the TTM learns a differentiable temporal shifting parameter ($\Delta$) for each input sequence. This module effectively aligns keyframes in time before they are processed by the classifier, providing temporal invariance without the need for recurrent architectures.

Experiments and Results

The model, referred to as 3s_net_TTM, was evaluated on three benchmark datasets: ChaLearn 2013, ChaLearn 2016 (Isolated), and MSRC-12.

Performance Metrics

The framework demonstrated superior accuracy across all datasets:

  • ChaLearn 2013: 92.08% accuracy, surpassing Two-stream LSTMs.
  • MSRC-12: 99.01% accuracy, establishing a new state-of-the-art.
  • ChaLearn 2016: 39.95% accuracy, outperforming other skeleton-only methods.

Computational Efficiency

The efficiency gains are a highlight of this work. On the ChaLearn 2013 dataset, the proposed model required only 4.69 million Multiply-Adds, compared to 358.34 million for a comparable Two-stream LSTM—a nearly 76x reduction in computational load.

Conclusion

By combining the mathematical rigor of Path Signatures with learned temporal alignment via the TTM, this work proves that efficient FC-based networks can outperform complex deep learning architectures in gesture recognition. Future work will focus on improving robustness to noisy skeleton data and incorporating features for static hand poses.

Summary updated

Abstract

The skeleton based gesture recognition is gaining more popularity due to its wide possible applications. The key issues are how to extract discriminative features and how to design the classification model. In this paper, we first leverage a robust feature descriptor, path signature (PS), and propose three PS features to explicitly represent the spatial and temporal motion characteristics, i.e., spatial PS (S PS), temporal PS (T PS) and temporal spatial PS (T S PS). Considering the significance of fine hand movements in the gesture, we propose an ”attention on hand” (AOH) principle to define joint pairs for the S PS and select single joint for the T PS. In addition, the dyadic method is employed to extract the T PS and T S PS features that encode global and local temporal dynamics in the motion. Secondly, without the recurrent strategy, the classification model still faces challenges on temporal variation among different sequences. We propose a new temporal transformer module (TTM) that can match the sequence key frames by learning the temporal shifting parameter for each input. This is a learning-based module that can be included into standard neural network architecture. Finally, we design a multi-stream fully connected layer based network to treat spatial and temporal features separately and fused them together for the final result. We have tested our method on three benchmark gesture datasets, i.e., ChaLearn 2016, ChaLearn 2013 and MSRC-12. Experimental results demonstrate that we achieve the state-of-the-art performance on skeleton-based gesture recognition with high computational efficiency.
BibTeX
@inproceedings{li2019skeleton,
  author = {Chenyang Li and Xin Zhang and Lufan Liao and Lianwen Jin and Weixin Yang},
  title = {{Skeleton-Based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module.}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33018585},
}
Figure