Hierarchical Deep Feature Learning For Decoding Imagined Speech From EEG

Pramit Saha, Sidney S. Fels

Abstract

We propose a mixed deep neural network strategy, incorporating a parallel combination of Convolutional (CNN) and Recurrent Neural Networks (RNN), cascaded with deep autoencoders and fully connected layers towards automatic identification of imagined speech from EEG. Instead of utilizing raw EEG channel data, we compute the joint variability of the channels in the form of a covariance matrix that provides spatio-temporal representations of EEG. The networks are trained hierarchically and the extracted features are passed onto the next network hierarchy until the final classification. Using a publicly available EEG-based speech imagery database, we demonstrate around 23.45% improvement of accuracy over the baseline method. Our approach demonstrates the promise of a mixed DNN approach for complex spatial-temporal classification problems.

The Challenge of Imagined Speech

Brain-Computer Interfaces (BCIs) aim to translate neural activity into external actions. "Imagined speech"—the internal articulation of words without physical movement—is a particularly complex target. Unlike motor imagery, speech involves high-dimensional coordination in the mind. EEG signals are notoriously noisy and subject-dependent, making it difficult for traditional models to distinguish between subtle phonetic differences.

The core innovation of this work is moving away from raw multi-channel EEG time series. Instead, the researchers focus on Channel Cross-Covariance (CCV). By analyzing how different electrodes fluctuate together, the model captures the information exchange between brain regions, which is more representative of the complex cognitive process of speech synthesis than raw voltage spikes.

A Hierarchical Deep Learning Framework

The proposed architecture employs a three-level hierarchy designed to mimic the brain's own processing of articulatory speech, which involves intermediate feedback loops and memory.

The proposed hierarchical architecture, showing the flow from EEG input to CCV matrices, parallel CNN/RNN feature extraction, DAE dimensionality reduction, and final FCN classification.
  1. Level 1: Parallel Feature Extraction: The CCV data is processed simultaneously by a 1D-CNN to extract spatial relationships and an RNN (using LSTMs) to capture temporal evolution.
  2. Level 2: Dimensionality Reduction: Features from both paths are concatenated and passed through a Deep Autoencoder (DAE). This unsupervised step compresses the data into a dense, discriminative latent vector.
  3. Level 3: Final Classification: A two-layered Fully Connected Network (FCN) with a softmax layer uses this latent vector to predict the specific vowel or word.

Results and Performance

The framework was evaluated using a dataset of 15 subjects performing tasks ranging from simple vowels (/a/, /i/, /u/) to short words ("in", "out") and long words ("cooperate", "independent").

The hierarchical approach significantly outperformed existing benchmarks, achieving an average 23.45% improvement in accuracy. Notably, the model showed remarkable consistency across all subjects, a common stumbling block for previous EEG-based models.

Comparison of classification accuracy across different subjects. The proposed approach (light blue) consistently outperforms baseline methods like Nguyen et al. and Log+LDA.

For the most challenging category—long words—the performance gains were particularly striking. For instance, for Subject 3 (S3), accuracy jumped from a baseline of 64.3% to 90.7%. This suggests that the hierarchical structure successfully identifies a more "universal" EEG manifold for speech processing.

Conclusion

This research demonstrates that a mixed DNN approach, combining spatial and temporal feature extraction with hierarchical refinement, is highly effective for decoding imagined speech. While future work is needed to address real-time processing constraints and broader generalization, this framework represents a significant step toward intuitive, speech-based BCIs.

Reference: Saha, P., & Fels, S. (2019). Hierarchical Deep Feature Learning For Decoding Imagined Speech From EEG. arXiv:1904.04352v1 [cs.LG].

Summary updated

Abstract

We propose a mixed deep neural network strategy, incorporating parallel combination of Convolutional (CNN) and Recurrent Neural Networks (RNN), cascaded with deep autoencoders and fully connected layers towards automatic identification of imagined speech from EEG. Instead of utilizing raw EEG channel data, we compute the joint variability of the channels in the form of a covariance matrix that provide spatio-temporal representations of EEG. The networks are trained hierarchically and the extracted features are passed onto the next network hierarchy until the final classification. Using a publicly available EEG based speech imagery database we demonstrate around 23.45% improvement of accuracy over the baseline method. Our approach demonstrates the promise of a mixed DNN approach for complex spatial-temporal classification problems.
BibTeX
@inproceedings{saha2019hierarchical,
  author = {Pramit Saha and Sidney S. Fels},
  title = {{Hierarchical Deep Feature Learning For Decoding Imagined Speech From EEG}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=YsnCH3l2p9},
}
Figure