Speech Synthesis Using EEG

Gautam Krishna, Co Tran, Yan Han, Mason Carnahan, Ahmed H. Tewfik

TL;DR

This paper explores synthesizing speech directly from electroencephalography (EEG) signals, a non-invasive brain-computer interface. By training a recurrent neural network (RNN) on EEG data recorded while subjects listened to or spoke specific phrases, the researchers achieved promising results in reconstructing speech features, paving the way for assistive technologies for individuals with speech impairments.

What Problem Does This Paper Address?

Communicating effectively is fundamental to human interaction. For individuals who have lost the ability to speak due to conditions like amyotrophic lateral sclerosis (ALS), restoring this capability is a critical challenge. While invasive methods like electrocorticography (ECoG) have shown promise in decoding speech from brain activity, non-invasive techniques are highly desirable for broader accessibility and safety. Electroencephalography (EEG) offers a non-invasive way to measure brain activity with high temporal resolution. However, effectively translating complex EEG signals into coherent speech has remained a significant hurdle. This paper aims to advance the field by demonstrating the feasibility of direct speech synthesis from non-invasively recorded EEG, building upon prior work in speech recognition from EEG.

Methods

The core of this research lies in a recurrent neural network (RNN) regression model designed to predict acoustic features directly from EEG signals. Specifically, the authors employed a gated recurrent unit (GRU) based RNN architecture. This model consists of two GRU layers with 256 and 128 hidden units, respectively, followed by a time-distributed dense layer that outputs 13 acoustic features at each time step. Dropout regularization was applied between layers to prevent overfitting.

The model was trained using EEG features as input and mel-frequency cepstral coefficients (MFCCs) as the target acoustic features. For feature extraction, EEG signals were filtered and processed using Independent Component Analysis (ICA) to remove artifacts. Three distinct EEG feature sets, previously introduced in related work, were utilized. For speech, MFCCs of dimension 13 were extracted. To further refine the EEG features and potentially reduce noise, Kernel Principal Component Analysis (KPCA) was applied for dimension reduction, with different dimensions retained for each feature set (30 for set 1, 50 for set 2, and 93 for set 3). The model was trained using the Adam optimizer with a learning rate of 0.01, a batch size of 100, and mean squared error (MSE) as the regression loss function.

Datasets

The study utilized a newly introduced EEG dataset consisting of EEG and speech signals recorded from four undergraduate UT Austin students. For each subject, EEG was recorded in parallel with both listening to and speaking specific utterances. The utterances used were common voice commands: "Hi Bixby," "Call Mom," "Open Camera," and "What's the weather." For each subject and each sentence, 70 speech-EEG recordings were collected. The data was split into 80% for training, 10% for validation, and 10% for testing.

Key Insights and Ideas

  • Feasibility of Non-Invasive Speech Synthesis: The paper strongly demonstrates that synthesizing speech directly from non-invasive EEG signals is achievable, offering a promising avenue for assistive communication technologies.
  • Impact of Data Quantity: The researchers observed that with a sufficient amount of training data, the choice of specific EEG feature sets had a less significant impact on the speech synthesis performance. This suggests that data augmentation and collection are crucial for robust EEG-based speech synthesis.
  • Subject-Specific Variability: Despite using the same utterances, significant variations in performance were observed across different subjects. This highlights the inherent individuality of brain signal generation for speech perception and production, suggesting that personalized models might be necessary for optimal results.
  • Effectiveness of GRU and Dropout: The use of a GRU-based RNN with dropout regularization led to improved speech synthesis performance compared to previous LSTM-based approaches, indicating the benefits of these architectural choices.

Key Results

The paper reports significant improvements in speech synthesis performance compared to prior work [1]. For synthesizing speech from EEG recorded in parallel with spoken speech, the authors achieved a Mel Cepstral Distortion (MCD) value as low as 0.433, a substantial improvement over the previously reported 5.737. Similarly, for EEG recorded during listening, their approach yielded an MCD of 0.471, outperforming the previous result of 1.34. These results were consistently observed across the four subjects, with specific MCD values detailed in the tables for both listening and spoken EEG conditions. For instance, for Subject 4 predicting listen MFCCs, MCD values were around 0.433-0.435 across different feature sets. For Subject 1 predicting spoken MFCCs, MCD values were around 0.841-0.856.

Limitations and Future Work

The authors acknowledge that their results represent a significant first step and that further research is needed. A key limitation is the relatively small number of subjects (four) and the limited number of utterances used. The observed subject-specific variations suggest that future work should focus on developing personalized models or techniques that can adapt to individual brain signal patterns. The paper also mentions that they did not perform experiments with Generative Adversarial Network (GAN) models due to challenges outlined in prior work, leaving this as a potential area for future exploration. The authors plan to publish the data sets used in this work to encourage further research in the field.

Summary updated

Abstract

In this paper we demonstrate speech synthesis using different electroencephalography (EEG) feature sets recently introduced in [1]. We make use of a recurrent neural network (RNN) regression model to predict acoustic features directly from EEG features. We demonstrate our results using EEG features recorded in parallel with spoken speech as well as using EEG recorded in parallel with listening utterances. We provide EEG based speech synthesis results for four subjects in this paper and our results demonstrate the feasibility of synthesizing speech directly from EEG features.
BibTeX
@inproceedings{krishna2020speech,
  author = {Gautam Krishna and Co Tran and Yan Han and Mason Carnahan and Ahmed H. Tewfik},
  title = {{Speech Synthesis Using EEG}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=gf8AwyxHkY},
}
Figure