Decoding inner speech with an end-to-end brain-to-text neural interface

Yizi Zhang, Linyang He, Chaofei Fan, Tingkai Liu, Han Yu, Trung Le, Jingyuan Li, Scott W. Linderman, Lea Duncker, Francis R. Willett, Nima Mesgarani, Liam Paninski

Abstract

Restoring communication for individuals with paralysis through speech Brain-Computer Interfaces (BCIs) has traditionally relied on cascaded models that predict phonemes before generating text. This paper introduces BIT (BraIn-to-Text), an end-to-end framework that translates neural activity directly into text. By employing cross-species pretraining on 367 hours of human and monkey neural data and utilizing audio-centric Large Language Models (LLMs), BIT achieves new state-of-the-art results. The framework significantly narrows the performance gap between attempted and imagined speech, offering a robust foundation model for neural decoding.

Introduction

Speech BCIs aim to decode neural signals into text to assist those with severe motor impairments. Traditional systems often use a cascaded approach, where a recurrent neural network (RNN) predicts phonemes that are subsequently processed by a separate language model. However, these disjointed stages are not jointly optimized, leading to potential performance bottlenecks. BIT addresses these challenges by creating a unified, differentiable system capable of leveraging large-scale, multi-species datasets to build a robust "foundation model" for the brain.

The BIT Framework

The BIT framework is built upon three core innovations:

  1. Cross-Species Pretraining: A transformer-based neural encoder is pretrained on 367 hours of neural recordings from both humans (~98 hours) and monkeys (~269 hours), covering both speech and motor tasks. This approach leverages universal primate neural dynamics.
  2. Audio-LLM Integration: The system integrates audio-augmented LLMs (such as Aero1-Audio). The authors hypothesize that neural speech signals align better with audio-trained models due to the biological links between motor and auditory systems.
  3. Modality Alignment: Using contrastive learning, the model projects neural embeddings and text embeddings into a shared latent space, ensuring matching brain-text pairs are closely aligned.
The BIT framework architecture, featuring cross-species pretraining (A), a transformer-based neural encoder (B), and an audio-augmented LLM decoder (C).

Experimental Results

BIT demonstrates significant improvements across multiple benchmarks. In the Brain-to-Text Benchmark '24, it reduced the Word Error Rate (WER) of the previous best end-to-end method from 24.69% to 10.22%. In a cascaded configuration, it achieved a record-low WER of 1.76% on the 2025 benchmark.

Crucially, for "imagined" inner speech—a notoriously difficult task—BIT provided a 39–45% improvement in accuracy compared to models trained from scratch. This suggests that the shared semantic structure learned during pretraining generalizes effectively from attempted speech to purely mental tasks.

Performance comparison of BIT against baselines for attempted (A) and imagined (B) speech, with qualitative decoding examples (D).

Discussion and Limitations

While BIT sets new performance records, several hurdles remain for clinical deployment. The end-to-end approach currently exhibits higher latency (~0.95s per sentence) compared to simpler cascaded models (~0.24s). Additionally, the model's use of bidirectional attention necessitates adaptation for real-time, causal applications. Future work will focus on model distillation for lower-power devices and addressing the non-stationarity of neural signals over time.

Conclusion

BIT represents a significant step toward high-performance, robust speech BCIs by treating neural activity as a foundational modality aligned with language and audio representations. By bridging the gap between attempted and imagined speech, it paves the way for more intuitive communication aids for individuals with severe paralysis.

Summary updated

BibTeX
@inproceedings{zhang2026decoding,
  author = {Yizi Zhang and Linyang He and Chaofei Fan and Tingkai Liu and Han Yu and Trung Le and Jingyuan Li and Scott W. Linderman and Lea Duncker and Francis R. Willett and Nima Mesgarani and Liam Paninski},
  title = {{Decoding inner speech with an end-to-end brain-to-text neural interface}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=Lp1noMpMUG},
}
Figure