Abstract
This paper presents a highly efficient framework for skeleton-based gesture recognition that leverages Path Signature (PS) features and a novel Temporal Transformer Module (TTM). By shifting the complexity of temporal modeling to the feature extraction phase, the proposed method replaces resource-intensive Recurrent Neural Networks (RNNs) and 3D Convolutional Neural Networks (3D-CNNs) with simple Fully Connected (FC) layers. The framework achieves state-of-the-art performance on major benchmarks while requiring significantly fewer computational resources—demonstrating a 76x reduction in Multiply-Adds compared to traditional LSTM-based approaches.
Introduction
Recognizing hand gestures from 3D skeleton data is a complex task due to significant intra-class variations in speed, duration, and execution style. While deep learning models like LSTMs and 3D-CNNs are effective at capturing temporal dependencies, they are computationally expensive and often difficult to train on smaller datasets. Conversely, hand-crafted features often lack the representational power needed for complex dynamics. This research proposes a "middle ground" by using Path Signatures—robust mathematical descriptors of trajectories—to represent motion, allowing for a lightweight classification model based on FC layers.
Methodology
System Overview
The proposed framework consists of three main components: the Attention on Hand (AOH) principle, the Path Signature (PS) extractor, and the Temporal Transformer Module (TTM).
Path Signature (PS) Features and AOH
Path Signatures are a collection of iterated integrals that provide a unique representation for paths. To manage the dimensionality of these features, the "Attention on Hand" (AOH) principle focuses on joints most relevant to gestures (elbows, wrists, fingers) and their relative positions to the rest of the body. This keeps the feature vector compact while focusing on the most discriminative parts of a gesture.
The framework extracts three types of PS features:
- Spatial PS (S_PS): Captures geometric relationships between joints within a single frame.
- Temporal PS (T_PS): Captures the evolution of individual joints over time.
- Temporal Spatial PS (T_S_PS): Captures how spatial correlations between joints evolve over time.
Temporal Transformer Module (TTM)
To handle variations in gesture speed, the TTM learns a differentiable temporal shifting parameter ($\Delta$) for each input sequence. This module effectively aligns keyframes in time before they are processed by the classifier, providing temporal invariance without the need for recurrent architectures.
Experiments and Results
The model, referred to as 3s_net_TTM, was evaluated on three benchmark datasets: ChaLearn 2013, ChaLearn 2016 (Isolated), and MSRC-12.
Performance Metrics
The framework demonstrated superior accuracy across all datasets:
- ChaLearn 2013: 92.08% accuracy, surpassing Two-stream LSTMs.
- MSRC-12: 99.01% accuracy, establishing a new state-of-the-art.
- ChaLearn 2016: 39.95% accuracy, outperforming other skeleton-only methods.
Computational Efficiency
The efficiency gains are a highlight of this work. On the ChaLearn 2013 dataset, the proposed model required only 4.69 million Multiply-Adds, compared to 358.34 million for a comparable Two-stream LSTM—a nearly 76x reduction in computational load.
Conclusion
By combining the mathematical rigor of Path Signatures with learned temporal alignment via the TTM, this work proves that efficient FC-based networks can outperform complex deep learning architectures in gesture recognition. Future work will focus on improving robustness to noisy skeleton data and incorporating features for static hand poses.
Summary updated