Abstract
Restoring communication for individuals with paralysis through speech Brain-Computer Interfaces (BCIs) has traditionally relied on cascaded models that predict phonemes before generating text. This paper introduces BIT (BraIn-to-Text), an end-to-end framework that translates neural activity directly into text. By employing cross-species pretraining on 367 hours of human and monkey neural data and utilizing audio-centric Large Language Models (LLMs), BIT achieves new state-of-the-art results. The framework significantly narrows the performance gap between attempted and imagined speech, offering a robust foundation model for neural decoding.
Introduction
Speech BCIs aim to decode neural signals into text to assist those with severe motor impairments. Traditional systems often use a cascaded approach, where a recurrent neural network (RNN) predicts phonemes that are subsequently processed by a separate language model. However, these disjointed stages are not jointly optimized, leading to potential performance bottlenecks. BIT addresses these challenges by creating a unified, differentiable system capable of leveraging large-scale, multi-species datasets to build a robust "foundation model" for the brain.
The BIT Framework
The BIT framework is built upon three core innovations:
- Cross-Species Pretraining: A transformer-based neural encoder is pretrained on 367 hours of neural recordings from both humans (~98 hours) and monkeys (~269 hours), covering both speech and motor tasks. This approach leverages universal primate neural dynamics.
- Audio-LLM Integration: The system integrates audio-augmented LLMs (such as Aero1-Audio). The authors hypothesize that neural speech signals align better with audio-trained models due to the biological links between motor and auditory systems.
- Modality Alignment: Using contrastive learning, the model projects neural embeddings and text embeddings into a shared latent space, ensuring matching brain-text pairs are closely aligned.
Experimental Results
BIT demonstrates significant improvements across multiple benchmarks. In the Brain-to-Text Benchmark '24, it reduced the Word Error Rate (WER) of the previous best end-to-end method from 24.69% to 10.22%. In a cascaded configuration, it achieved a record-low WER of 1.76% on the 2025 benchmark.
Crucially, for "imagined" inner speech—a notoriously difficult task—BIT provided a 39–45% improvement in accuracy compared to models trained from scratch. This suggests that the shared semantic structure learned during pretraining generalizes effectively from attempted speech to purely mental tasks.
Discussion and Limitations
While BIT sets new performance records, several hurdles remain for clinical deployment. The end-to-end approach currently exhibits higher latency (~0.95s per sentence) compared to simpler cascaded models (~0.24s). Additionally, the model's use of bidirectional attention necessitates adaptation for real-time, causal applications. Future work will focus on model distillation for lower-power devices and addressing the non-stationarity of neural signals over time.
Conclusion
BIT represents a significant step toward high-performance, robust speech BCIs by treating neural activity as a foundational modality aligned with language and audio representations. By bridging the gap between attempted and imagined speech, it paves the way for more intuitive communication aids for individuals with severe paralysis.
Summary updated