Coupling-Graph Channel Embeddings for Label-Free Calibration of Intracortical CTC Decoders

Speech and handwriting brain-computer interfaces decode spikes recorded on a grid of roughly a hundred implanted electrodes into text. They are trained with Connectionist Temporal Classification (CTC), a loss that lets a network learn to emit a character or phoneme sequence from a much longer input stream without knowing when each symbol occurred. These decoders tie each weight to a specific electrode index. Electrode indices mean nothing across people, and much less than one would hope across days. Every new user therefore needs a trainable input layer fit on labeled sentences, and even then cross-person transfer lags behind what the same brain achieves with scrambled electrodes. We propose to describe each electrode by its role in the population's coupling network rather than by its index. That network is inferred from unlabeled spiking with a pairwise maximum entropy model: the least-committal probability distribution over binary population patterns that still reproduces each channel's firing rate and every pairwise co-firing rate. The decoder's input layer then becomes a shared function of these roles. A new user could be calibrated from minutes of spontaneous activity, with no transcribed sentences.

The idea in two sentences: fit an Ising-style coupling matrix to a new user's unlabeled threshold crossings, and run a small graph network over it to give every electrode a position-free embedding. A CTC decoder trained on other users reads spikes through these embeddings, so it can be applied to the new user before any labeled data exists and fine-tuned after.

Core Claim

Two bodies of evidence motivate this. Pairwise maximum entropy models capture most of the structure in multi-neuron firing. They account for roughly 99% of structured activity in primate retina (The Structure of Multi-Neuron Firing Patterns in Primate Retina, The Structure of Large-Scale Synchronized Firing in Primate Retina), and they predict collective hippocampal configurations from low-order statistics (Collective Behavior of Place and Non-place Neurons in the Hippocampal Network). They have not been tested as a front end for sequence decoding.

On the decoding side, Cross-brain transfer of high-performance intracortical speech and handwriting BCIs trains five-layer GRU decoders with CTC across BrainGate2 participants T5, T12, T15, T16 and T18. It finds that session-specific linear input layers are mandatory and that transfer only helps below about 200 target sentences. Even with those layers, cross-brain transfer falls short of within-brain electrode-permuted controls for some participants, including T12 and T15 speech. A Generalist Intracortical Motor Decoder reports the same "sensor variability wall" at 350M parameters and 2,000 hours of data.

Our claim is narrower than "physics helps." The coupling matrix $J$ separates direct pairwise dependence from rate effects and from shared drive that each unit's field absorbs. It is computable without labels, and its graph structure does not depend on electrode order. A permutation-equivariant input layer conditioned on $J$ should therefore reach a given phoneme error rate with fewer labeled sentences than an index-tied linear layer, and it should narrow the gap between cross-brain transfer and the electrode-permuted control.

This differs from all existing CTC work in the file. EnCTC (Connectionist Temporal Classification with Maximum Entropy Regularization) and AdaMER-CTC (AdaMER-CTC) apply maximum entropy to the distribution over alignment paths, not to the input population. CTC-CRF (CAT) adds pairwise couplings between adjacent output labels in time, not between input channels. We leave the CTC loss untouched and put the maximum entropy model where the neuroscience evidence says it belongs: on the electrodes.

Research Program

Coupling estimation. Threshold crossings are binned at 20 ms and binarized, so $\sigma_i \in \{0,1\}$ indicates whether electrode $i$ crossed threshold in a bin. For each session we fit

$$P(\vec\sigma)=\frac{1}{Z}\exp\Big(\sum_i h_i\sigma_i+\sum_{i<j}J_{ij}\sigma_i\sigma_j\Big),$$

where $h_i$ is electrode $i$'s bias, $J_{ij}$ its symmetric coupling to electrode $j$, and $Z$ the normalizer. Because $Z$ is intractable for 96 to 256 channels, we fit by L1-regularized pseudo-likelihood, which conditions each unit on all the others and avoids sampling.

Three findings constrain this step. Strong couplings can make samplers non-ergodic (Bistability, non-ergodicity, and inhibition in pairwise maximum-entropy models), so we avoid relying on sampling. Cortical Ising parameters carry real estimation uncertainty (Ensemble inhibition and excitation in the human cortex), so we bootstrap $J$ and feed the decoder resampled couplings during training as augmentation. Random subsampling of a population produces spurious collective signatures (Signatures of criticality arise from random subsampling), so we make no criticality claims and treat $J$ only as a descriptor.

We also check whether the pairwise model is adequate for motor-cortex threshold crossings at all. On random 20-channel subsets, where $Z$ can be computed exactly, we measure the fraction of multi-information captured, following the retina analyses.

Decoder. Each electrode receives an embedding $e_i$. A three-layer message-passing graph network computes it from that electrode's bias, its rate, and its weighted neighbors in $J$. The first-layer representation of frame $t$ is $\sum_i x_{t,i}\,\phi(e_i)$, where $x_{t,i}$ is the binned count on electrode $i$ and $\phi$ is a shared network. This is invariant to electrode order by construction. A GRU-plus-CTC backbone identical to the cross-brain paper's sits on top.

Training and calibration. We train jointly on source participants. For a new participant, $J$ is fit on unlabeled data, either inter-trial periods or a few minutes of rest. The decoder is evaluated zero-shot and then fine-tuned on 10, 20, 50, 100 and 200 labeled sentences.

Baselines

The naive reference is a participant-specific GRU-CTC decoder trained from scratch on the target's labeled sentences alone. This is the floor transfer must beat at every label budget.

The strongest existing method on this question is the cross-brain transfer recipe of Cross-brain transfer of high-performance intracortical speech and handwriting BCIs: a shared stacked GRU trained with CTC, plus a trainable affine input layer per session. The file gives qualitative results but no reported phoneme error rate for cross-brain transfer at fixed sentence counts. The only figure available is a saturation case: T18 reaches 2% CER with about 20 sentences, which is why we focus on lower-signal participants. We will rerun this recipe ourselves, with the same participants, splits, backbone size, optimizer and hyperparameter search budget as our model. The only difference will be the input layer.

On the single-participant premotor brain-to-text corpus, the strongest reported number is Teaching Wav2Vec2 the Language of the Brain. Full fine-tuning there reports 18.54% CER and 30.97% WER without a language model, against 39.01% CER from scratch. We treat this as a cross-session reference rather than a cross-brain one, since that corpus has one participant.

The key ablations hold the graph network fixed and swap the graph it reads:

  • the Pearson correlation matrix of the binarized data;
  • a Gaussian precision matrix, the continuous analogue of $J$;
  • an empty graph, so only rates and biases are used;
  • $J$ with its rows shuffled.

Beating the affine-layer baseline means lower phoneme error rate at 10 to 50 sentences, with confidence intervals over seeds and held-out sessions that exclude zero. It also means a measurably smaller gap to the electrode-permuted within-brain control. Beating the correlation and precision ablations is what licenses the specific maximum entropy claim.

Why not just…?

Why not just use the trainable affine input layer the cross-brain paper already uses? It needs labeled sentences to fit, so it cannot help at zero labels. The paper itself shows it fails to bridge structural differences between brains: cross-brain transfer lags the electrode-permuted control for several participants. Our label-budget curves compare directly against it. If it matches us at 20 sentences, the advantage is confined to the zero-shot regime, and we will report it that way.

Why not just build the graph from raw correlations or a precision matrix? This is the real threat. Correlations are cheaper and may carry most of the same information. Our hypothesis is that during attempted speech, correlations are dominated by shared task-driven rate changes, which in the Ising model are absorbed into the fields $h_i$, so $J$ should transfer better. The ablation settles it. If correlation or precision graphs match $J$ within seed noise, we will conclude that a second-moment coupling graph is what matters and that maximum entropy adds nothing. That would be a useful negative result, and the method would survive under a smaller claim.

Why not just scale a foundation model? NDT3 (A Generalist Intracortical Motor Decoder) shows that scaling alone does not remove the sensor variability wall, and that scratch models catch up after about 1.5 hours of task data. Our input layer is complementary to scale. It could be placed under any backbone, and we test it under the same GRU so the comparison isolates the input layer.

Why not just add EnCTC or AdaMER path-entropy regularization? Those methods address overconfident alignments, not electrode identity, and they leave the input layer index-tied. We include EnCTC as an orthogonal add-on to both our model and the baseline, to show that any gain from coupling embeddings survives it.

Proposed Experiments

Experiments use the BrainGate2 speech and handwriting data described in the cross-brain paper. We run leave-one-participant-out on T5, T12, T15 and T16, and report T18 only as a saturation check. The premotor brain-to-text corpus is used for cross-session drift: train on early days and calibrate later days from unlabeled data alone.

Metrics are phoneme error rate for speech and character error rate for handwriting, plotted against labeled-sentence count from 0 to 200. We also report the gap between cross-brain transfer and the electrode-permuted control.

Three checks are decisive. First, zero-label calibration must beat chance-level decoding and the empty-graph ablation on every held-out participant. Second, $J$-based embeddings must beat correlation-based embeddings, which is what tests the maximum entropy claim specifically. Third, the pairwise model must capture a substantial fraction of multi-information on exact 20-channel subsets. If it does not, the scientific premise fails for motor cortex, and we will report that before interpreting any decoding gains.