A Reservoir Computing Approach for Human Gesture Recognition from Kinect Data

Claudio Gallicchio, Alessio MicheliAI*AAL@AI*IA 2016 · OpenReview

Introduction

Human gesture recognition is a fundamental component of Ambient Assisted Living (AAL) and natural user interfaces. While sensors like the Microsoft Kinect provide high-fidelity 3D skeletal data, interpreting these signals is traditionally difficult due to variations in speed, duration, and execution style across different individuals. Conventional approaches often rely on labor-intensive manual feature engineering—such as calculating joint angles or velocities—before applying classifiers like Support Vector Machines (SVMs) or Hidden Markov Models (HMMs).

This research explores a more efficient alternative: Reservoir Computing (RC). By utilizing bidirectional Echo State Networks (ESNs), the researchers demonstrate that raw 3D trajectories can be processed directly, bypassing the feature engineering bottleneck while achieving state-of-the-art accuracy.

The Reservoir Computing Paradigm

The core of this approach is the Echo State Network (ESN). Unlike standard Recurrent Neural Networks (RNNs) where all connections are trained via backpropagation—a process often plagued by vanishing gradients and high computational costs—an ESN employs a large, fixed, and untrained recurrent "reservoir." This reservoir maps input signals into a high-dimensional state space. Only a simple linear "readout" layer is trained to map these states to the desired output classes.

Bidirectional Architecture and Mean State Mapping

To capture the full temporal context of a gesture, the authors implemented Bidirectional Leaky Integrator ESNs (LI-ESNs). This architecture processes the gesture sequence simultaneously from "left-to-right" (forward in time) and "right-to-left" (backward in time).

Architecture of the bidirectional Reservoir Computing model for gesture recognition.

Because gestures vary in length, the model employs Mean State Mapping. The reservoir states are averaged over the entire duration of the sequence to create a fixed-size vector. This vector is then fed into a Multi-Layer Perceptron (MLP) with a softmax layer to determine class probabilities. Furthermore, the reservoir's internal weights are initialized using a permutation matrix to push the network toward a "critical regime," optimizing its memory and processing capabilities.

Experimental Setup

The model was evaluated using the AALTD2016 challenge dataset. The input consisted of 3D trajectories from 8 body joints (hands, elbows, wrists, and thumbs), resulting in a 24-dimensional input at each time step. The dataset included 6 distinct human gestures, with a balanced split of 180 training sequences and 180 blind test sequences.

One of the most significant findings was the model's ability to handle noisy, raw 3D coordinates without human-defined geometric features. By simply rescaling the data to a [-1, 1] range, the reservoir effectively extracted the necessary temporal patterns.

Results and Discussion

The proposed architecture demonstrated exceptional performance. While a single RC network achieved high accuracy, the researchers utilized an ensemble of 30 networks—each with different random reservoir initializations—to improve robustness.

Validation and Test set accuracy for the proposed model.

The ensemble reached a 94.4% accuracy on the blind test set, earning 5th place overall in the AALTD2016 challenge. Notably, the model achieved 100% training accuracy, indicating that the reservoir was highly effective at separating gesture classes in the high-dimensional state space.

Conclusion

This work highlights the efficiency and efficacy of Reservoir Computing for temporal pattern recognition. By avoiding complex feature engineering and training only the readout layer, the model offers a computationally inexpensive solution for gesture recognition.

However, the bidirectional approach currently requires the entire gesture sequence to be completed before processing, making it less suitable for real-time streaming applications. Future work may focus on adapting this architecture for real-time recognition and scaling the system to larger, more diverse gesture libraries. Given its training efficiency, RC is also a strong candidate for systems that must adapt quickly to new users or changing environments in the field.

Summary updated

BibTeX
@inproceedings{gallicchio2016reservoir,
  author = {Claudio Gallicchio and Alessio Micheli},
  title = {{A Reservoir Computing Approach for Human Gesture Recognition from Kinect Data}},
  booktitle = {AI*AAL@AI*IA},
  year = {2016},
  url = {https://openreview.net/forum?id=do5d4SqQkY},
}
Figure