Abstract
We propose a mixed deep neural network strategy, incorporating parallel combination of Convolutional (CNN) and Recurrent Neural Networks (RNN), cascaded with deep autoencoders and fully connected layers towards automatic identification of imagined speech from EEG. Instead of utilizing raw EEG channel data, we compute the joint variability of the channels in the form of a covariance matrix that provide spatio-temporal representations of EEG. The networks are trained hierarchically and the extracted features are passed onto the next network hierarchy until the final classification. Using a publicly available EEG based speech imagery database we demonstrate around 23.45% improvement of accuracy over the baseline method. Our approach demonstrates the promise of a mixed DNN approach for complex spatialtemporal classification problems.
Introduction
Brain-Computer Interfaces (BCIs) represent a transformative technology designed to translate neural activity into external commands or text. While significant strides have been made in decoding motor imagery (such as imagining hand movements), the decoding of imagined speech—the internal phonation of words without vocalization—remains a formidable challenge.
EEG signals associated with speech imagery are characterized by low signal-to-noise ratios, high dimensionality, and significant inter-session variability. Furthermore, speech is a complex cognitive process requiring rapid spatio-temporal coordination across multiple brain regions. Traditional methods often rely on manual feature engineering, which fails to capture the intricate patterns of the brain's "internal monologue." This paper introduces a hierarchical deep learning approach to overcome these limitations.
Methodology: Hierarchical Mixed DNN Architecture
The proposed framework moves away from processing raw EEG data, which is often too noisy for direct deep learning applications. Instead, it employs a three-level hierarchical strategy.
1. Input Transformation: Channel Cross-Covariance (CCV)
Rather than feeding raw electrode voltages into the network, we compute the Channel Cross-Covariance (CCV). This transformation generates a matrix representing the joint variability between different electrodes over time, effectively capturing the functional connectivity and information transfer between brain regions during speech synthesis.
2. Level 1: Parallel Feature Extraction
The CCV matrices are processed by two parallel networks to capture distinct signal characteristics:
- 1D-CNN: Optimized for extracting spatial features and identifying critical connections between electrodes.
- RNN (with LSTMs): Designed to capture the latent temporal dependencies and the sequential "memory" inherent in the EEG signal.
3. Level 2: Dimensionality Reduction via DAE
The features extracted by the CNN and RNN are concatenated and passed through a Deep Autoencoder (DAE). This unsupervised learning stage compresses the high-dimensional feature vector into a dense, latent representation, effectively filtering out non-informative noise and stripping away session-specific artifacts.
4. Level 3: Classification
The final stage utilizes a Fully Connected Network (FCN) with a Softmax layer. This classifier takes the compressed latent vector to predict the specific vowel or word being imagined.
Experimental Setup and Results
The model was evaluated using a publicly available imagined speech EEG dataset featuring 15 healthy subjects. The vocabulary included vowels (/a/, /i/, /u/), short words ('in', 'out', 'up'), and long words ('cooperate', 'independent').
Key Findings
The hierarchical mixed DNN approach significantly outperformed existing state-of-the-art methods across the board:
- Overall Accuracy: The proposed model achieved a 23.45% improvement in overall accuracy compared to the baseline method.
- Complex Word Decoding: The model showed exceptional performance on multi-syllabic words. For instance, for Subject 3 (S3), accuracy for long words improved from a previous best of 64.3% to 90.7%.
- Stability: Unlike previous models that often specialized in either vowels or words, this architecture maintained consistent performance across all vocabulary types.
Discussion and Future Work
The success of this approach stems from two primary innovations. First, the use of Covariance over Raw Data allows the model to focus on the brain's functional connectivity rather than raw voltage fluctuations. Second, the Mixed Architecture respects both the spatial ("where") and temporal ("when") dimensions of neural activity.
Despite these advancements, several challenges remain. The current vocabulary is limited; scaling to natural language remains a significant hurdle. Additionally, the computational overhead of calculating covariance matrices must be optimized for real-time BCI applications. Future research will also need to address cross-subject generalization to enable systems that do not require extensive individual retraining.
Summary updated