Transfer learning via distributed brain recordings enables reliable speech decoding

Aditya Singh, Tessy M. Thomas, Jinlong Li, Gregory Hickok, Xaq Pitkow, Nitin TandonNature Communications 2025 · DOI

Abstract: Speech brain-computer interfaces (BCIs) combine neural recordings with large language models to achieve real-time intelligible speech. However, these decoders rely on dense, intact cortical coverage and are challenging to scale across individuals with heterogeneous brain organization. To derive scalable transfer learning strategies for neural speech decoding, we used minimally invasive stereo-electroencephalography recordings in a large cohort performing a demanding speech motor task. A sequence-to-sequence model enabled decoding of variable-length phonemic sequences prior to and during articulation. This enabled development of a cross-subject transfer learning framework to isolate shared latent manifolds while enabling individual model initialization. The group-derived decoder significantly outperformed models trained on individual data alone, enabling decoding robustness despite variable coverage and activation. These results highlight a pathway toward generalizable neural prostheses for speech and language disorders by leveraging large-scale intracranial datasets with distributed spatial sampling and shared task demands.

Introduction

Current high-performance speech brain-computer interfaces (BCIs) typically rely on "bespoke" models. These systems require dense electrode arrays, such as electrocorticography (ECoG) grids, placed directly onto an intact sensorimotor cortex. While effective, this approach faces significant hurdles in scalability due to inter-individual brain heterogeneity and the clinical reality that many patients needing BCIs have damage to the primary motor areas.

This study, published in Nature Communications, explores a more resilient and generalizable approach. By utilizing a large-scale dataset and transfer learning, researchers demonstrated that speech can be decoded reliably even when brain recordings are sparse or distributed across different regions. The core innovation lies in a transfer learning framework that identifies "latent manifolds"—shared patterns of neural activity—across a diverse cohort, moving BCIs toward universal neural prostheses.

Experimental Design and Methodology

The researchers employed stereo-electroencephalography (sEEG), which uses minimally invasive depth electrodes to provide a distributed view of the brain's speech network. The study involved 25 patients with intractable epilepsy, utilizing over 3,600 electrodes in total.

Experimental task design, electrode coverage across the patient cohort, and resulting brain activation maps.

Participants performed a demanding "tongue twister" task involving 128 distinct stimuli covering 36 English phonemes. This task was designed to heavily load the articulatory system, generating rich neural signals for decoding. The decoding architecture utilized a Sequence-to-Sequence (Seq2Seq) model:

  1. Feature Extraction: A 1D Convolutional layer extracts features from raw neural data.
  2. Encoding: An RNN (LSTM) Encoder maps these features into a latent state representing intended speech.
  3. Decoding: An RNN Decoder translates the latent state into phoneme sequences.

The "secret sauce" of this approach is transfer learning. The model was pre-trained on a "source" group to learn universal speech dynamics in the recurrent layers. When applied to a new "target" subject, only the input and output layers were fine-tuned, allowing the model to leverage shared neural logic.

Key Results and Insights

The study yielded several critical findings regarding the universality and robustness of speech signals:

  • Universal Latent Manifolds: The recurrent layers of the model, which handle the temporal dynamics of speech, proved highly transferable. This suggests that the fundamental neural program for speech production is remarkably consistent across individuals.
  • Robustness to Sparse Coverage: Unlike single-subject models that fail if the motor cortex is occluded, multi-subject group models remained resilient. They successfully extracted speech information from various "hubs" within the distributed language network.
  • Pre-Articulatory Decoding: The system could decode phonemes during the planning phase, before articulation began. This is vital for reducing latency in real-time BCI applications.
  • Performance Gains: The model achieved a median Phoneme Error Rate (PER) of 27% during articulation. Crucially, transfer learning significantly improved performance for subjects with limited data, reducing PER from 57% (within-subject) to 45% (transfer model) in specific tests.

Discussion and Future Directions

While these results represent a major step toward generalizable BCIs, several challenges remain. The current study focused on "overt" speech in patients with functional vocal abilities. Future work must validate these models for "covert" or "inner" speech in anarthric patients who cannot speak. Additionally, while phoneme-based decoding is flexible, expanding the task from structured tongue twisters to natural, open-vocabulary conversation is the next frontier for the field.

By leveraging large-scale intracranial datasets and shared task demands, this research highlights a viable pathway toward scalable and clinically robust neural prostheses for speech and language disorders.

Source: Singh, A., et al. (2025). Transfer learning via distributed brain recordings enables reliable speech decoding. Nature Communications. https://doi.org/10.1038/s41467-025-63825-0

Summary updated

Abstract

Speech brain-computer interfaces (BCIs) combine neural recordings with large language models to achieve real-time intelligible speech. However, these decoders rely on dense, intact cortical coverage and are challenging to scale across individuals with heterogeneous brain organization. To derive scalable transfer learning strategies for neural speech decoding, we used minimally invasive stereo-electroencephalography recordings in a large cohort performing a demanding speech motor task. A sequence-to-sequence model enabled decoding of variable-length phonemic sequences prior to and during articulation. This enabled development of a cross-subject transfer learning framework to isolate shared latent manifolds while enabling individual model initialization. The group-derived decoder significantly outperformed models trained on individual data alone, enabling decoding robustness despite variable coverage and activation. These results highlight a pathway toward generalizable neural prostheses for speech and language disorders by leveraging large-scale intracranial datasets with distributed spatial sampling and shared task demands.
BibTeX
@inproceedings{singh2025transfer,
  author = {Aditya Singh and Tessy M. Thomas and Jinlong Li and Gregory Hickok and Xaq Pitkow and Nitin Tandon},
  title = {{Transfer learning via distributed brain recordings enables reliable speech decoding}},
  booktitle = {Nature Communications},
  year = {2025},
  doi = {10.1038/s41467-025-63825-0},
}
Figure