Hierarchical Deep Feature Learning for Decoding Imagined Speech from EEG

Pramit Saha, Sidney S. Fels

Abstract

We propose a mixed deep neural network strategy, incorporating parallel combination of Convolutional (CNN) and Recurrent Neural Networks (RNN), cascaded with deep autoencoders and fully connected layers towards automatic identification of imagined speech from EEG. Instead of utilizing raw EEG channel data, we compute the joint variability of the channels in the form of a covariance matrix that provide spatio-temporal representations of EEG. The networks are trained hierarchically and the extracted features are passed onto the next network hierarchy until the final classification. Using a publicly available EEG based speech imagery database we demonstrate around 23.45% improvement of accuracy over the baseline method. Our approach demonstrates the promise of a mixed DNN approach for complex spatialtemporal classification problems.

Introduction

Brain-Computer Interfaces (BCIs) represent a transformative technology designed to translate neural activity into external commands or text. While significant strides have been made in decoding motor imagery (such as imagining hand movements), the decoding of imagined speech—the internal phonation of words without vocalization—remains a formidable challenge.

EEG signals associated with speech imagery are characterized by low signal-to-noise ratios, high dimensionality, and significant inter-session variability. Furthermore, speech is a complex cognitive process requiring rapid spatio-temporal coordination across multiple brain regions. Traditional methods often rely on manual feature engineering, which fails to capture the intricate patterns of the brain's "internal monologue." This paper introduces a hierarchical deep learning approach to overcome these limitations.

Methodology: Hierarchical Mixed DNN Architecture

The proposed framework moves away from processing raw EEG data, which is often too noisy for direct deep learning applications. Instead, it employs a three-level hierarchical strategy.

1. Input Transformation: Channel Cross-Covariance (CCV)

Rather than feeding raw electrode voltages into the network, we compute the Channel Cross-Covariance (CCV). This transformation generates a matrix representing the joint variability between different electrodes over time, effectively capturing the functional connectivity and information transfer between brain regions during speech synthesis.

2. Level 1: Parallel Feature Extraction

The CCV matrices are processed by two parallel networks to capture distinct signal characteristics:

  • 1D-CNN: Optimized for extracting spatial features and identifying critical connections between electrodes.
  • RNN (with LSTMs): Designed to capture the latent temporal dependencies and the sequential "memory" inherent in the EEG signal.
The proposed hierarchical mixed DNN architecture for imagined speech decoding.

3. Level 2: Dimensionality Reduction via DAE

The features extracted by the CNN and RNN are concatenated and passed through a Deep Autoencoder (DAE). This unsupervised learning stage compresses the high-dimensional feature vector into a dense, latent representation, effectively filtering out non-informative noise and stripping away session-specific artifacts.

4. Level 3: Classification

The final stage utilizes a Fully Connected Network (FCN) with a Softmax layer. This classifier takes the compressed latent vector to predict the specific vowel or word being imagined.

Experimental Setup and Results

The model was evaluated using a publicly available imagined speech EEG dataset featuring 15 healthy subjects. The vocabulary included vowels (/a/, /i/, /u/), short words ('in', 'out', 'up'), and long words ('cooperate', 'independent').

Key Findings

The hierarchical mixed DNN approach significantly outperformed existing state-of-the-art methods across the board:

  • Overall Accuracy: The proposed model achieved a 23.45% improvement in overall accuracy compared to the baseline method.
  • Complex Word Decoding: The model showed exceptional performance on multi-syllabic words. For instance, for Subject 3 (S3), accuracy for long words improved from a previous best of 64.3% to 90.7%.
  • Stability: Unlike previous models that often specialized in either vowels or words, this architecture maintained consistent performance across all vocabulary types.
Accuracy comparison across subjects for different decoding methods.

Discussion and Future Work

The success of this approach stems from two primary innovations. First, the use of Covariance over Raw Data allows the model to focus on the brain's functional connectivity rather than raw voltage fluctuations. Second, the Mixed Architecture respects both the spatial ("where") and temporal ("when") dimensions of neural activity.

Despite these advancements, several challenges remain. The current vocabulary is limited; scaling to natural language remains a significant hurdle. Additionally, the computational overhead of calculating covariance matrices must be optimized for real-time BCI applications. Future research will also need to address cross-subject generalization to enable systems that do not require extensive individual retraining.

Summary updated

Abstract

We propose a mixed deep neural network strategy, incorporating parallel combination of Convolutional (CNN) and Recurrent Neural Networks (RNN), cascaded with deep autoencoders and fully connected layers towards automatic identification of imagined speech from EEG. Instead of utilizing raw EEG channel data, we compute the joint variability of the channels in the form of a covariance matrix that provide spatio-temporal representations of EEG. The networks are trained hierarchically and the extracted features are passed onto the next network hierarchy until the final classification. Using a publicly available EEG based speech imagery database we demonstrate around 23.45% improvement of accuracy over the baseline method. Our approach demonstrates the promise of a mixed DNN approach for complex spatialtemporal classification problems.
BibTeX
@inproceedings{saha2019hierarchical,
  author = {Pramit Saha and Sidney S. Fels},
  title = {{Hierarchical Deep Feature Learning for Decoding Imagined Speech from EEG}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=YsnCH3l2p9},
}
Figure