Rapid Decoding of Hand Gestures in Electrocorticography Using Recurrent Neural Networks

Gang Pan, Jiajun Li, Yu Qi, Hang Yu, Junming Zhu, Xiaoxiang Zheng, Yueming Wang, Shaomin ZhangFrontiers in Neuroscience 2018 · DOI

TL;DR

This paper introduces a novel Recurrent Neural Network (RNN) approach for decoding hand gestures from electrocorticography (ECoG) signals, achieving high accuracy by effectively leveraging temporal dynamics. The method demonstrates rapid gesture recognition within 0.5 seconds of motion onset, paving the way for more responsive brain-computer interfaces (BCIs) for prosthetic control.

What Problem Does This Paper Address?

Brain-computer interfaces (BCIs) offer a promising avenue for restoring motor function in individuals with disabilities. ECoG signals, recorded directly from the brain's surface, are rich in information about motor intentions. However, a significant challenge in using ECoG for hand gesture decoding has been the reliance on long time windows in existing methods, which fail to capture the crucial temporal dynamics of gestures. This oversight limits the speed and responsiveness of BCIs. This paper aims to address this by developing a decoder that can effectively utilize the temporal patterns within ECoG signals for faster and more accurate hand gesture recognition.

Methods

The core innovation of this work lies in the application of Recurrent Neural Networks (RNNs), specifically Long Short-Term Memory (LSTM) networks, for decoding hand gestures from ECoG signals. The proposed method involves several key steps:

  • Segmentation: ECoG signals are divided into short temporal segments (300 ms window with 100 ms stride).
  • Feature Extraction: Power Spectral Density (PSD) is calculated for each segment and normalized using baseline ECoG data from a "relax" state. These PSD values are then aggregated into five frequency bands (low-frequency, beta, low gamma, high gamma, and high frequency).
  • Feature Selection: A greedy strategy is employed to select the most informative channels and frequency bands. This iterative process selects features that maximally improve decoding accuracy when combined with already selected features, aiming to reduce noise and computational cost.
  • RNN-based Decoding: The selected features from sequential temporal segments are fed into an LSTM network. The LSTM's ability to model temporal dependencies allows it to learn and exploit the dynamic patterns of hand gestures within the ECoG time series.

The use of RNNs is a departure from traditional methods that often flatten temporal features into a single vector, thereby losing sequential information.

Datasets

The study utilized ECoG data collected from two participants (P1 and P2) who had implanted intracranial electrode arrays for epilepsy surgery. The data was collected during a behavioral task where participants performed three hand gestures: "scissors," "rock," and "paper." The paper does not introduce a new benchmark dataset but rather uses data from these two individuals for evaluation.

Key Insights and Ideas

  • Temporal Dynamics are Crucial: The study strongly emphasizes that the temporal patterns within ECoG signals are highly informative for accurate and rapid gesture decoding, a factor often overlooked by methods using long time windows.
  • RNNs Excel at Temporal Modeling: LSTM networks proved effective in capturing the sequential nature of ECoG data, leading to superior performance compared to methods that treat segments independently or flatten temporal information.
  • Greedy Feature Selection is Effective: The proposed greedy feature selection strategy efficiently identifies informative channels and frequency bands, improving decoding performance and reducing computational load without significant loss of accuracy.
  • High-Frequency Bands are Key: The analysis revealed that higher frequency bands (70-135 Hz and 135-200 Hz) in ECoG signals are particularly correlated with hand movements.
  • Rapid Recognition is Achievable: The research demonstrates that gestures can be recognized with high accuracy in very short time intervals after motion onset, a critical factor for real-time BCI applications.

Key Results

  • Overall Accuracy: The RNN-based decoder achieved an impressive average accuracy of 90% for decoding three hand gestures across the two participants.
  • Rapid Recognition: The system demonstrated the ability to recognize gestures within 0.5 seconds after motion onset with an accuracy of approximately 80%. This is a significant improvement over many existing methods that require longer detection delays.
  • Feature Selection Performance: The greedy feature selection method achieved performance close to using all features but with a significantly reduced feature set, highlighting its efficiency.
  • Comparison with Other Methods: The proposed RNN method statistically outperformed linear regression (LR-Segments), Support Vector Machines (SVM-Segments), and Multilayer Perceptrons (MLP-Segments) in gesture recognition accuracy.

Limitations and Future Work

  • Limited Number of Participants: The study was conducted on only two participants. While the results are promising, a larger and more diverse participant pool would be necessary to generalize the findings.
  • Overfitting Concerns: The authors noted that with small datasets, using too many features could lead to overfitting, and they opted for a smaller set of the most informative features to mitigate this.
  • Future Work: The authors suggest that further exploration of temporal information in ECoG signals is warranted. They also highlight the potential for their method in online BCI control of prosthetic and robotic devices, implying future work in real-time implementation and user studies. The paper also implicitly suggests that further optimization of feature selection and RNN architectures could lead to even better performance.

Summary updated

Abstract

Brain-computer interface (BCI) is a direct communication pathway between brain and external devices, and BCI-based prosthetic devices are promising to provide new rehabilitation options for people with motor disabilities. Electrocorticography (ECoG) signals contain rich information correlated with motor activities, and have great potential in hand gesture decoding. However, most existing decoders use long time windows, thus ignore the temporal dynamics within the period. In this study, we propose to use recurrent neural networks (RNNs) to exploit the temporal information in ECoG signals for robust hand gesture decoding. With RNN's high nonlinearity modeling ability, our method can effectively capture the temporal information in ECoG time series for robust gesture recognition. In the experiments, we decode three hand gestures using ECoG signals of two participants, and achieve an accuracy of 90%. Specially, we investigate the possibility of recognizing the gestures in a time interval as short as possible after motion onsets. Our method rapidly recognizes gestures within 0.5 s after motion onsets with an accuracy of about 80%. Experimental results also indicate that the temporal dynamics is especially informative for effective and rapid decoding of hand gestures.
BibTeX
@inproceedings{pan2018rapid,
  author = {Gang Pan and Jiajun Li and Yu Qi and Hang Yu and Junming Zhu and Xiaoxiang Zheng and Yueming Wang and Shaomin Zhang},
  title = {{Rapid Decoding of Hand Gestures in Electrocorticography Using Recurrent Neural Networks}},
  booktitle = {Frontiers in Neuroscience},
  year = {2018},
  doi = {10.3389/fnins.2018.00555},
}
Figure