Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction

Jihwan Lee, Tiantian Feng, Aditya Kommineni, Sudarsana Reddy Kadiri, Shrikanth Narayanan

Abstract

Brain-computer interfaces (BCI) offer numerous human-centered application possibilities, particularly affecting people with neurological disorders. Text or speech decoding from brain activities is a relevant domain that could augment the quality of life for people with impaired speech perception. We propose a novel approach to enhance listened speech decoding from electroencephalography (EEG) signals by utilizing an auxiliary phoneme predictor that simultaneously decodes textual phoneme sequences. The proposed model architecture consists of three main parts: EEG module, speech module, and phoneme predictor. The EEG module learns to properly represent EEG signals into EEG embeddings. The speech module generates speech waveforms from the EEG embeddings. The phoneme predictor outputs the decoded phoneme sequences in text modality. Our proposed approach allows users to obtain decoded listened speech from EEG signals in both modalities (speech waveforms and textual phoneme sequences) simultaneously, eliminating the need for a concatenated sequential pipeline for each modality. The proposed approach also outperforms previous methods in both modalities.

Introduction

Brain-Computer Interfaces (BCIs) aim to restore communication for individuals with neurological or speech disorders by decoding brain activity into text or audio. However, EEG signals are notoriously noisy and complex, making high-fidelity decoding a significant challenge. Most current research focuses on a single output: either reconstructing the acoustic waveform (speech) or predicting the text (phonemes). Systems requiring both typically rely on concatenated pipelines—for instance, decoding text first and then running it through a separate speech synthesizer. Such approaches are inefficient and prone to error accumulation. Furthermore, single-task models often struggle to extract robust, linguistically meaningful representations from raw EEG.

Methodology

The proposed framework decodes speech and text simultaneously from a shared EEG representation using a multi-task architecture.

Overview of the proposed parallel decoding architecture, featuring the EEG Module, Speech Module, and Phoneme Predictor.

The architecture consists of three main components:

  1. EEG Module: An encoder-decoder structure where the encoder uses Structured State Space Sequence (S4) blocks to transform raw EEG into latent embeddings. The decoder reconstructs the original EEG signal as a self-supervised task to stabilize training.
  2. Speech Module: This module generates a speech waveform from EEG embeddings using a non-causal WaveNet architecture and a HiFi-GAN vocoder.
  3. Phoneme Predictor: An auxiliary task using Conformer blocks followed by an LSTM decoder to predict phoneme sequences, trained with Connectionist Temporal Classification (CTC) loss.

A gradient stop is implemented between the EEG and speech modules to ensure the encoder's learning is driven primarily by phoneme prediction and EEG reconstruction, preserving linguistic integrity in the embeddings.

Datasets and Evaluation

The study utilized the N400 dataset (128-channel EEG from 24 subjects listening to 440 English sentences). Evaluation focused on three scenarios: Unseen Speech, Unseen Subject, and "Unseen Both" (zero-shot).

Results and Discussion

The inclusion of a phoneme predictor acts as a powerful regularizer, helping the EEG encoder focus on linguistic structures while ignoring noise.

Performance metrics (MCD, Mel-Corr, and Top-k Accuracy) across different phonetic categories and model configurations (CB0, CB1, CB2).

Key results include:

  • Speech Quality: In the most challenging "unseen both" category, the model achieved a Mel-Correlation of 28.34%, more than doubling the previous state-of-the-art (13.05%).
  • Distortion: Mel-cepstral distortion (MCD) was reduced from 11.65 dB to 10.29 dB.
  • Phoneme Accuracy: The model outperformed the EEG2TEXT baseline in Top-3 and Top-5 accuracy.

The researchers observed a trade-off with the number of Conformer blocks (CB-0, CB-1, CB-2). While more blocks (CB-2) improved phoneme accuracy, they slightly degraded speech reconstruction quality, suggesting a specialization effect where acoustic nuances are lost in favor of textual precision.

Conclusion

By decoding in parallel, this system eliminates the need for separate synthesizers, reducing latency for real-time BCI applications. While currently focused on listened speech perception, future work aims to apply this framework to imagined or attempted speech. Additionally, integrating large-scale pre-trained models and expanding the vocabulary remain key areas for development.

Summary updated

Abstract

Brain-computer interfaces (BCI) offer numerous human-centered application possibilities, particularly affecting people with neurological disorders. Text or speech decoding from brain activities is a relevant domain that could augment the quality of life for people with impaired speech perception. We propose a novel approach to enhance listened speech decoding from electroencephalography (EEG) signals by utilizing an auxiliary phoneme predictor that simultaneously decodes textual phoneme sequences. The proposed model architecture consists of three main parts: EEG module, speech module, and phoneme predictor. The EEG module learns to properly represent EEG signals into EEG embeddings. The speech module generates speech waveforms from the EEG embeddings. The phoneme predictor outputs the decoded phoneme sequences in text modality. Our proposed approach allows users to obtain decoded listened speech from EEG signals in both modalities (speech waveforms and textual phoneme sequences) simultaneously, eliminating the need for a concatenated sequential pipeline for each modality. The proposed approach also outperforms previous methods in both modalities. The source code and speech samples are publicly available.
BibTeX
@inproceedings{lee2025enhancing,
  author = {Jihwan Lee and Tiantian Feng and Aditya Kommineni and Sudarsana Reddy Kadiri and Shrikanth Narayanan},
  title = {{Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=auNmyTELGC},
}
Figure