A speech brain-computer interface (BCI) records spikes from a small electrode array implanted in motor cortex and turns them into text. The strongest results come from one person at a time. One participant with a pontine stroke, using a single 64-channel array, reached a median 19.6% word error rate on a 125,000-word vocabulary (Restoring brain-to-text communication…). Like a clinical case report, such a result proves that something is possible. It cannot say how often it will happen in other patients. There is also a further confound: the headline error rate mixes two sources of information. One is the neural evidence from a few dozen electrodes. The other is a 6.7-billion-parameter language model that already knows which sentences are likely. In fMRI and EEG, separating these two collapsed apparent decoding performance, with BLEU-1 dropping from about 50.24 to about 22.83 on Narratives (Rethinking Cross-Subject Data Splitting…). Nobody has done this accounting for intracortical spike decoding.
We propose to measure, for a single participant, how many bits per phoneme the neural recording actually contributes beyond what the language model and sentence memorization already supply. We will then test whether that number is a more portable summary of an N-of-1 result than word error rate. "Portable" here means it better predicts performance on new sentences, on degraded arrays with fewer channels, and on other days. In two sentences: we treat each single-patient BCI result as a case report whose signal must be separated from its priors, and we build the controls that do this separation. If they work, small-array results become comparable across patients and labs in a way word error rate is not.
Core Claim
Word error rate after language-model rescoring is a poor quantity to generalize from in single-participant intracortical BCIs. It saturates and hides how much neural information a low-channel array carries. A leakage-controlled, null-referenced measure of neural information per phoneme is better behaved in three ways. First, it declines smoothly as electrodes are removed, whereas word error rate stays flat and then fails abruptly. Second, it predicts accuracy on unscripted, never-seen sentences better than standard-split word error rate does. Third, it is stable across days in a way that tracks recording quality rather than vocabulary.
Two pieces are new relative to Rethinking Cross-Subject Data Splitting…. That paper diagnoses leakage through how data are split, and only for non-invasive recordings. We add the invasive, low-channel regime, and we add a "null brain" reference that measures what the decoder achieves when the neural input is uninformative.
The Measurement
The decoders of interest are recurrent networks trained with connectionist temporal classification, a loss that scores a phoneme sequence against a neural time series without needing frame-level alignment. These are the five-layer GRU decoders with participant-specific input layers used in Cross-brain transfer…. This loss gives a log-likelihood for the true phoneme sequence, so we can define
$$ I = \frac{1}{|y|}\Big[\log p_\theta(y \mid x) - \log p_\theta(y \mid \tilde{x})\Big], $$
where $y$ is the true phoneme sequence of a held-out sentence, $|y|$ is its length, $x$ is the real neural recording, $\tilde{x}$ is a null recording, and $p_\theta$ is the decoder trained on real data. The null recording comes from a different trial of matched length, time-shuffled within channel. It preserves firing rates and channel statistics but carries no information about $y$. We compute $I$ only on a sentence-disjoint split, where no test sentence's text appears in training. This rules out the possibility that the recurrent network itself has memorized frequent sentences.
We report the same contrast at the word level after decoding with the language model. The word error rate obtained from null recordings is the "language-model floor." The gap between the floor and the real word error rate is the neural contribution in interpretable units.
Research Program
The work has three stages.
The first stage builds the evaluation splits for the public human intracortical speech dataset distributed with the Willett et al. 2023 study. That dataset was used by Latent Diffusion for Neural Spiking Data, and its trials run 2 to 10 seconds. We adapt the subject, story, segment and signal graph of the Rethinking paper to intracortical sessions. This yields signal-leakage and text-leakage rates for the standard split. It also yields three clean alternatives: a sentence-disjoint split, a day-disjoint split, and a split that is both sentence- and day-disjoint.
The second stage trains the reference decoder on each split. We use the GRU with connectionist temporal classification and day-specific input layers. Decoding uses a 5-gram model followed by large-language-model rescoring, mirroring the pipeline in Restoring brain-to-text communication…. For each split we compute $I$, the language-model floor, phoneme error rate, and word error rate.
The third stage asks which summary generalizes. We run three tests:
- Electrode-dropout curves. We randomly subsample the array from its full channel count down to a handful of channels. This directly simulates lower-throughput hardware and tells a single-case study how much array it really needed.
- Prediction of held-out difficulty. We test whether each summary predicts performance on the hardest held-out material: novel sentences, and the free-response trials if the dataset contains them.
- Day-to-day tracking. We test whether each summary follows across-day drift that the field currently corrects with roughly 36 calibration sentences, or about 6 minutes, per day.
If BrainGate collaborators share data from additional participants, the same pipeline extends without change into a small cross-participant check of whether $I$ ranks participants in the order of their transfer benefit. That benefit is reported in Cross-brain transfer…, where transfer helped only when target data was below 200 sentences. The core claims do not depend on this extension.
Prior work plays three roles here:
- Motivation. The leakage results of Rethinking Cross-Subject Data Splitting… motivate the work. So does the case-report literature's insistence that an N=1 outcome may reflect the patient rather than the method. Laparoscopic pancreaticoduodenectomy in A-92-older Chinese patient… attributes a surgical success to the patient's unusual performance status. The BCI analogue is attributing a low word error rate to the language model or to an unusually high-signal implant.
- Constraint. Cross-brain transfer… found that cross-brain transfer underperforms within-user electrode permutation for some participants. That finding warns that a single scalar may not capture differences in neural geometry. We therefore report $I$ alongside, not instead of, error rates.
- Method. The decoder architecture and calibration conventions come from the BrainGate papers.
The longitudinal deployment in A Personalized and Adaptable User Interface… also shapes the work. It reports a jump from about 40% to 59% fully correct sentences once user correction is added. This reminds us that real-world accuracy has a third, human-correction component, which we explicitly exclude from scope.
Baselines
Naive reference: the headline metric. The do-nothing reference is the field's current practice: word error rate with language-model rescoring on the dataset's standard split. For context, the closest reported figures are a median 19.6% word error rate on the 125,000-word vocabulary and 10.0% on the 1,024-word vocabulary for participant T16 (Restoring brain-to-text communication…). However, we found no reported result on Lacuna for word error rate on the Willett 2023 dataset itself. We will therefore reproduce that number with our pipeline rather than quote one.
Strongest existing method: the full decoding pipeline. The strongest competitor is the GRU decoder with connectionist temporal classification, participant- and day-specific affine input layers, 5-gram decoding and large-language-model rescoring. This is the pipeline behind every high-performing result in the file. All variants will use identical architecture, training steps and hyperparameter search budget, re-run per split. This ensures that differences come from the split and the metric, not from tuning.
Phoneme error rate as a competing summary. Phoneme error rate without a language model is the obvious competing summary. We compare it to $I$ on equal footing.
Ablation of the null reference. This ablation isolates the key ingredient by computing $I$ against a weaker null: a constant mean-rate input rather than trial-shuffled recordings. If the two nulls give the same rankings, the more elaborate null is unnecessary.
What counts as success. $I$ beats the alternatives if it predicts held-out novel-sentence accuracy, measured by rank correlation across electrode-dropout levels and days, significantly better than both standard-split word error rate and sentence-disjoint phoneme error rate.
Why not just…?
Why not just report phoneme error rate without the language model? This is the most serious alternative. Phoneme error rate removes the explicit language model but not the implicit one. On a standard split, the recurrent decoder can learn frequent sentence fragments and will look accurate on repeated text. That is exactly the text-stimulus leakage documented in Rethinking Cross-Subject Data Splitting…. Phoneme error rate is also a discrete count that saturates near zero for high-signal participants, such as T18 at about 2% character error rate after 20 sentences in Cross-brain transfer…. If sentence-disjoint phoneme error rate turns out to predict held-out difficulty as well as $I$, we will say so and recommend it as the cheaper standard.
Why not just use sentence-disjoint splits and stop there? Clean splits are half of our method, and they may account for most of the effect. They still leave the reader unable to tell how much of the remaining accuracy the language model would deliver from an uninformative brain. The language-model floor answers that question, and it matters most for small vocabularies where the prior is strong. The split-only variant is one of our conditions, so the data will show whether the null reference adds anything.
Why not just report within-subject calibration curves? A calibration curve, error versus number of training sentences, is a natural portable summary. The T16 result that 36 sentences reach 95% of peak is an example. But the curve inherits whatever leakage and language-model contribution the endpoint metric has, so it does not resolve the confound. We will compute calibration curves for both word error rate and $I$ and report which curve shape is more consistent across days.
Decisive Checks
Three results would settle the claim.
How large is the leakage gap? We measure the gap between standard-split and sentence-disjoint word error rate on the Willett data. If the gap is negligible, intracortical decoding is largely free of the memorization problem seen in fMRI. That is a useful negative result and a direct answer to Gap 1.
How high is the language-model floor? If null recordings decoded through the 125,000-word pipeline produce word error rates far below 100%, headline numbers credit the brain with prior knowledge. The floor quantifies how much.
How do the metrics behave as electrodes are removed? Under the core claim, word error rate stays near its full-array value across the first several dropout levels and then rises sharply. Over the same range, $I$ falls roughly monotonically. Rank correlation with held-out novel-sentence accuracy should be higher for $I$.
If instead word error rate degrades smoothly and predicts held-out accuracy as well as $I$, the proposal's central claim is wrong. In that case, the current reporting convention for single-patient BCI studies is adequate as it stands.
References
- Restoring brain-to-text communication in a person with dysarthria from pontine stroke using an intracortical brain-computer interface
- Rethinking Cross-Subject Data Splitting for Brain-to-Text Decoding
- Cross-brain transfer of high-performance intracortical speech and handwriting BCIs
- Latent Diffusion for Neural Spiking Data
- Laparoscopic pancreaticoduodenectomy in A-92-older Chinese patient for cancer of head of the pancreas
- A Personalized and Adaptable User Interface for a Speech and Cursor Brain-Computer Interface