Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction

Jihwan Lee, Tiantian Feng, Aditya Kommineni, Sudarsana Reddy Kadiri, Shrikanth Narayanan

TL;DR

Researchers have developed a new framework that decodes both speech audio and textual phoneme sequences directly from EEG signals in parallel. By incorporating an auxiliary phoneme predictor, the model not only provides two ways to "read" brain activity but also significantly improves the quality of the reconstructed speech audio compared to existing single-modality methods.

What Problem Does This Paper Address?

Brain-Computer Interfaces (BCIs) aim to restore communication for individuals with neurological or speech disorders by "decoding" brain activity into text or audio. However, EEG signals are notoriously noisy and complex.

Most current research focuses on a single output: either reconstructing the acoustic waveform (speech) or predicting the text (phonemes). If a system needs both, it typically uses a "concatenated pipeline"—for example, decoding text first and then running it through a separate speech synthesizer. This is inefficient and prone to error accumulation. Furthermore, single-task models often struggle to extract robust, linguistically meaningful representations from raw EEG.

Methods

The authors propose a multi-task architecture that decodes speech and text simultaneously from a shared EEG representation. The framework consists of three main components:

  1. EEG Module: An encoder-decoder structure. The encoder uses Structured State Space Sequence (S4) blocks to transform raw EEG into latent embeddings, while the decoder learns to reconstruct the original EEG signal (a self-supervised task that helps stabilize training).
  2. Speech Module: This module takes the EEG embeddings and generates a speech waveform. It utilizes a non-causal WaveNet architecture and a HiFi-GAN vocoder to produce high-fidelity audio.
  3. Phoneme Predictor: This is the "auxiliary" task. It uses Conformer blocks (which combine the benefits of Transformers and CNNs) followed by an LSTM decoder to predict phoneme sequences. It is trained using Connectionist Temporal Classification (CTC) loss, a standard in speech recognition for mapping variable-length sequences.

A key technical detail is the use of a gradient stop between the EEG and speech modules. This ensures that the EEG encoder's learning is driven primarily by the phoneme prediction and EEG reconstruction tasks, preventing the speech generation task from "polluting" the linguistic embeddings.

Datasets

The study utilized the N400 dataset, which contains 128-channel EEG data from 24 subjects. Participants listened to 440 synthesized, gender-neutral English sentences. The researchers evaluated the model on three distinct challenges:

  • Unseen Speech: Testing on sentences the model never heard during training.
  • Unseen Subject: Testing on brain data from people the model never saw.
  • Unseen Both: The most difficult "zero-shot" scenario, where both the person and the sentence are new to the model.

Key Insights and Ideas

  • Phonemes as a Regularizer: The most striking insight is that forcing the model to predict phonemes actually makes it better at "hearing" the audio. The phoneme predictor acts as a guide, helping the EEG encoder ignore noise and focus on the linguistic structures hidden in the brain waves.
  • The Conformer Trade-off: The researchers experimented with different numbers of Conformer blocks (CB-0, CB-1, CB-2). They found a performance trade-off: while more blocks improved phoneme accuracy, they slightly degraded the quality of the reconstructed speech. This suggests that as the model becomes more specialized in "textual" decoding, it may lose some of the fine-grained acoustic nuances required for audio reconstruction.
  • Parallelism vs. Pipelines: By decoding in parallel, the system eliminates the need for a separate synthesizer, making it a more viable candidate for real-time BCI applications where latency is critical.

Key Results

The proposed model (specifically the CB-0 configuration) significantly outperformed previous state-of-the-art baselines:

  • Speech Quality: In the "unseen both" category, the model achieved a Mel-Correlation of 28.34%, more than doubling the performance of the previous FESDE baseline (13.05%).
  • Distortion: It reduced Mel-cepstral distortion (MCD) from 11.65 dB to 10.29 dB, indicating much clearer audio reconstruction.
  • Phoneme Accuracy: The model outperformed the EEG2TEXT baseline, particularly in Top-3 and Top-5 accuracy metrics, showing it is more robust at predicting the correct sequence of sounds even in difficult "unseen" scenarios.

Limitations and Future Work

  • Perception vs. Production: This study focused on listened speech (how the brain processes incoming audio). The authors acknowledge that the next big hurdle is applying this to imagined or attempted speech, which is the ultimate goal for paralyzed users.
  • Model Scale: While the framework is novel, it does not yet leverage massive pre-trained models (like wav2vec 2.0 or large language models). The authors plan to integrate these to further boost performance.
  • Vocabulary Constraints: The current analysis is limited to the vocabulary and phoneme distribution of the N400 dataset; scaling to open-vocabulary, natural conversation remains a challenge.

Summary updated

Abstract

Brain-computer interfaces (BCI) offer numerous human-centered application possibilities, particularly affecting people with neurological disorders. Text or speech decoding from brain activities is a relevant domain that could augment the quality of life for people with impaired speech perception. We propose a novel approach to enhance listened speech decoding from electroencephalography (EEG) signals by utilizing an auxiliary phoneme predictor that simultaneously decodes textual phoneme sequences. The proposed model architecture consists of three main parts: EEG module, speech module, and phoneme predictor. The EEG module learns to properly represent EEG signals into EEG embeddings. The speech module generates speech waveforms from the EEG embeddings. The phoneme predictor outputs the decoded phoneme sequences in text modality. Our proposed approach allows users to obtain decoded listened speech from EEG signals in both modalities (speech waveforms and textual phoneme sequences) simultaneously, eliminating the need for a concatenated sequential pipeline for each modality. The proposed approach also outperforms previous methods in both modalities. The source code and speech samples are publicly available1.
BibTeX
@inproceedings{lee2025enhancing,
  author = {Jihwan Lee and Tiantian Feng and Aditya Kommineni and Sudarsana Reddy Kadiri and Shrikanth Narayanan},
  title = {{Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=auNmyTELGC},
}

Loading related content...

Figure