Abstract
Neuromorphic vision sensors offer a novel, event-driven sensing modality with advantages like low energy consumption and high dynamic range. However, traditional methods often convert sparse event streams into frames, losing these benefits. This paper proposes Fixed Length Gists Representation (FLGR) learning using a mixture density autoencoder to preserve the nature of event-based data. Combined with an RNN-HMM hybrid architecture, this approach addresses continuous gesture recognition. We also introduce the Neuro ConGD Dataset, featuring 17 gesture classes, to benchmark this high-efficiency sequence classification task.
Introduction
Traditional frame-based cameras capture images at fixed intervals, often leading to motion blur and data redundancy. Neuromorphic vision sensors, such as the Dynamic Vision Sensor (DVS), transmit only pixel-level changes (events) as they occur. While this provides microsecond resolution, many existing algorithms still accumulate events into artificial frames to utilize standard computer vision techniques. This "frame-based" approach negates the low-latency and sparse-data advantages of the sensor. Furthermore, real-world applications require continuous recognition—identifying gestures from an unsegmented stream while ignoring "blank" periods—rather than just classifying isolated clips.
Proposed Methodology: FLGR and RNN-HMM Hybrid
The proposed system utilizes a two-stage pipeline designed to handle event data natively without frame accumulation.
Representation Learning (FLGR)
Instead of creating frames, the researchers employ a Mixture Density Autoencoder (MDAE). This unsupervised model uses an encoder and decoder composed of Gated Recurrent Units (GRUs). The encoder compresses variable-length event sequences into a fixed-length vector, termed a "gist." This FLGR "gist" captures the statistical essence of the movement, providing a format that is easy for subsequent sequence classifiers to process while maintaining the sensor's inherent advantages.
Temporal Classification and Decoding
The classification stage uses a hybrid approach to ensure both local accuracy and global temporal consistency:
- RNN Classifier: A 3-layer GRU network processes the FLGR sequences. At each time step, it outputs a probability distribution across 17 classes (16 gestures plus a "blank" class).
- HMM Smoothing: To address "noisy" or flickering outputs from the RNN, a Hidden Markov Model (HMM) is employed. Using the Viterbi algorithm, the HMM finds the most likely sequence of gestures, effectively filtering out spurious detections and smoothing the results.
- Segmentation: An additional HMM layer distinguishes between "gesture" and "blank" states, allowing the system to localize exactly when a movement occurs within the continuous stream.
The Neuro ConGD Dataset
To evaluate the system, the authors developed the Neuro ConGD Dataset, a new benchmark for neuromorphic continuous hand gesture recognition. It contains 2,040 instances of 17 gesture classes (e.g., swipe-left, zoom-in, beckon) recorded with a DVS sensor. Unlike isolated clip datasets, Neuro ConGD provides continuous recordings where gestures are performed in random order with varying "blank" intervals, reflecting real-world conditions.
Experimental Results
The FLGR + RNN-HMM system demonstrates significant improvements over baseline methods:
- Accuracy: The proposed system achieved a Mean Jaccard Index of 86.9%, compared to 63.3% for a baseline using raw events and a standard RNN.
- Detection Performance: The system reached an F-score of 96.6% for the task of detecting gesture occurrences within the stream.
- Representation Efficiency: Using FLGR improved accuracy by over 15% compared to feeding raw event data, validating the effectiveness of the learned "gists."
Discussion and Future Work
The research highlights that image-level representations are not strictly necessary for understanding motion. By treating events as a continuous distribution, the model preserves the speed and power efficiency of neuromorphic sensors.
However, limitations remain. Very slow movements generate few events, leading to potential information loss. Future work may involve implementing bidirectional RNNs to leverage future context or using "rolling" hidden states in the autoencoder to better capture slow transitions and pauses within gestures.
Summary updated