FLGR: Fixed Length Gists Representation Learning for RNN-HMM Hybrid-Based Neuromorphic Continuous Gesture Recognition

Guang Chen, Jieneng Chen, Marten Lienen, Jörg Conradt, Florian Röhrbein, Alois KnollFrontiers in Neuroscience 2019 · DOI

Abstract

Neuromorphic vision sensors offer a novel, event-driven sensing modality with advantages like low energy consumption and high dynamic range. However, traditional methods often convert sparse event streams into frames, losing these benefits. This paper proposes Fixed Length Gists Representation (FLGR) learning using a mixture density autoencoder to preserve the nature of event-based data. Combined with an RNN-HMM hybrid architecture, this approach addresses continuous gesture recognition. We also introduce the Neuro ConGD Dataset, featuring 17 gesture classes, to benchmark this high-efficiency sequence classification task.

Introduction

Traditional frame-based cameras capture images at fixed intervals, often leading to motion blur and data redundancy. Neuromorphic vision sensors, such as the Dynamic Vision Sensor (DVS), transmit only pixel-level changes (events) as they occur. While this provides microsecond resolution, many existing algorithms still accumulate events into artificial frames to utilize standard computer vision techniques. This "frame-based" approach negates the low-latency and sparse-data advantages of the sensor. Furthermore, real-world applications require continuous recognition—identifying gestures from an unsegmented stream while ignoring "blank" periods—rather than just classifying isolated clips.

Proposed Methodology: FLGR and RNN-HMM Hybrid

The proposed system utilizes a two-stage pipeline designed to handle event data natively without frame accumulation.

Representation Learning (FLGR)

Instead of creating frames, the researchers employ a Mixture Density Autoencoder (MDAE). This unsupervised model uses an encoder and decoder composed of Gated Recurrent Units (GRUs). The encoder compresses variable-length event sequences into a fixed-length vector, termed a "gist." This FLGR "gist" captures the statistical essence of the movement, providing a format that is easy for subsequent sequence classifiers to process while maintaining the sensor's inherent advantages.

Overview of the FLGR and RNN-HMM hybrid pipeline for continuous gesture recognition.

Temporal Classification and Decoding

The classification stage uses a hybrid approach to ensure both local accuracy and global temporal consistency:

  1. RNN Classifier: A 3-layer GRU network processes the FLGR sequences. At each time step, it outputs a probability distribution across 17 classes (16 gestures plus a "blank" class).
  2. HMM Smoothing: To address "noisy" or flickering outputs from the RNN, a Hidden Markov Model (HMM) is employed. Using the Viterbi algorithm, the HMM finds the most likely sequence of gestures, effectively filtering out spurious detections and smoothing the results.
  3. Segmentation: An additional HMM layer distinguishes between "gesture" and "blank" states, allowing the system to localize exactly when a movement occurs within the continuous stream.
Architecture of the 3-layer GRU network used for sequence classification.

The Neuro ConGD Dataset

To evaluate the system, the authors developed the Neuro ConGD Dataset, a new benchmark for neuromorphic continuous hand gesture recognition. It contains 2,040 instances of 17 gesture classes (e.g., swipe-left, zoom-in, beckon) recorded with a DVS sensor. Unlike isolated clip datasets, Neuro ConGD provides continuous recordings where gestures are performed in random order with varying "blank" intervals, reflecting real-world conditions.

Experimental Results

The FLGR + RNN-HMM system demonstrates significant improvements over baseline methods:

  • Accuracy: The proposed system achieved a Mean Jaccard Index of 86.9%, compared to 63.3% for a baseline using raw events and a standard RNN.
  • Detection Performance: The system reached an F-score of 96.6% for the task of detecting gesture occurrences within the stream.
  • Representation Efficiency: Using FLGR improved accuracy by over 15% compared to feeding raw event data, validating the effectiveness of the learned "gists."

Discussion and Future Work

The research highlights that image-level representations are not strictly necessary for understanding motion. By treating events as a continuous distribution, the model preserves the speed and power efficiency of neuromorphic sensors.

However, limitations remain. Very slow movements generate few events, leading to potential information loss. Future work may involve implementing bidirectional RNNs to leverage future context or using "rolling" hidden states in the autoencoder to better capture slow transitions and pauses within gestures.

Summary updated

Abstract

A neuromorphic vision sensors is a novel passive sensing modality and frameless sensors with several advantages over conventional cameras. Frame-based cameras have an average frame-rate of 30 fps, causing motion blur when capturing fast motion, e.g., hand gesture. Rather than wastefully sending entire images at a fixed frame rate, neuromorphic vision sensors only transmit the local pixel-level changes induced by the movement in a scene when they occur. This leads to advantageous characteristics, including low energy consumption, high dynamic range, a sparse event stream and low response latency. In this study, a novel representation learning method was proposed: Fixed Length Gists Representation (FLGR) learning for event-based gesture recognition. Previous methods accumulate events into video frames in a time duration (e.g., 30 ms) to make the accumulated image-level representation. However, the accumulated-frame-based representation waives the friendly event-driven paradigm of neuromorphic vision sensor. New representation are urgently needed to fill the gap in non-accumulated-frame-based representation and exploit the further capabilities of neuromorphic vision. The proposed FLGR is a sequence learned from mixture density autoencoder and preserves the nature of event-based data better. FLGR has a data format of fixed length, and it is easy to feed to sequence classifier. Moreover, an RNN-HMM hybrid was proposed to address the continuous gesture recognition problem. Recurrent neural network (RNN) was applied for FLGR sequence classification while hidden Markov model (HMM) is employed for localizing the candidate gesture and improving the result in a continuous sequence. A neuromorphic continuous hand gestures dataset (Neuro ConGD Dataset) was developed with 17 hand gestures classes for the community of the neuromorphic research. Hopefully, FLGR can inspire the study on the event-based highly efficient, high-speed, and high-dynamic-range sequence classification tasks.
BibTeX
@inproceedings{chen2019flgr,
  author = {Guang Chen and Jieneng Chen and Marten Lienen and Jörg Conradt and Florian Röhrbein and Alois Knoll},
  title = {{FLGR: Fixed Length Gists Representation Learning for RNN-HMM Hybrid-Based Neuromorphic Continuous Gesture Recognition}},
  booktitle = {Frontiers in Neuroscience},
  year = {2019},
  doi = {10.3389/fnins.2019.00073},
}
Figure