Brain signals recorded while people listen to speech can be split, band by band, into two parts using the Hilbert transform (a standard operation that turns a filtered signal into a slowly varying loudness contour, the envelope, and a fast oscillation whose timing is carried by its phase, often called the temporal fine structure). Decoders built on both parts work well when trained and tested on the same people. They collapse when applied to someone new. The covert-speech study by Functional Connectivity and Hilbert-Based Features for Covert Speech EEG Variability Analysis and Classification reports near-random accuracy for direct cross-subject transfer. Most EEG cohorts have only 15 to 25 people, so this failure is also a sample-size problem: there are too few brains to learn around individual differences.
Our hypothesis is that much of this failure comes from something simple. Each person's brain responds to sound with a slightly different delay. A delay leaves the envelope nearly unchanged but rotates the phase in every band by an amount proportional to that band's frequency. If this holds, a per-person correction with only one or two numbers, estimated from about a minute of calibration listening, should make phase features transfer across people. It should also cut the number of training subjects a decoder needs. We propose to test this directly by measuring how decoding on unseen subjects improves as the number of training subjects grows, comparing envelope features, phase features, and delay-aligned phase features.
Core Claim
The claim has two parts, each falsifiable. First, the gap between within-subject and cross-subject decoding is concentrated in the phase channel, and it is concentrated there because of a per-subject delay. The phase offset between a subject's band-limited response and a group template should therefore grow linearly with the band's center frequency. Second, correcting only that delay should make phase features generalize across subjects roughly as well as envelope features do. Once the correction is applied, combining envelope and phase should reach a given accuracy on held-out subjects with fewer training subjects than either the uncorrected combination or the envelope alone.
Concretely, let $z_{s,c,b}(t)$ be the analytic signal (the complex signal whose magnitude is the envelope and whose angle is the phase) of subject $s$, channel $c$, band $b$ with center frequency $\omega_b$. The aligned signal is
$$\tilde z_{s,c,b}(t) = z_{s,c,b}(t)\, e^{-j(\omega_b \tau_{s,c} + \phi_{s,c})},$$
where $\tau_{s,c}$ is a subject- and channel-specific latency and $\phi_{s,c}$ is a constant phase offset. Both are fit by maximizing stimulus-locked coherence between $\tilde z$ and a leave-that-subject-out group template during a short calibration recording. This adds two parameters per channel per subject, shared across all bands. The linear-in-frequency form follows from the fact that a pure time shift $\tau$ multiplies each frequency component by $e^{-j\omega\tau}$. Because the model is this constrained, it should be estimable from seconds to minutes of data, whereas a free subject-specific input layer would need far more.
Relation to Prior Work
Three papers motivate the gap. The covert-speech study Functional Connectivity and Hilbert-Based Features for Covert Speech EEG Variability Analysis and Classification combines Hilbert envelope and fine-structure features with a recurrent classifier. It reaches 59.14% ± 4.32% only when all subjects are pooled in cross-validation, and near-random accuracy on truly new subjects. Because it concatenates the two feature types, it never shows which one breaks. Temporal envelope and fine structure cues for dysarthric speech detection using CNNs does separate them, on acoustic speech rather than EEG. It finds envelope-only AUC 0.83 and fine-structure-only AUC 0.72, versus 0.93 combined. This tells us phase carries complementary information worth rescuing, but it says nothing about transfer across people. Neural Entrainment to Natural Speech Envelope Based on Subject Aligned EEG Signals averages EEG from 22 listeners and doubles delta-band envelope reconstruction from r ≈ 0.25 to 0.50. The authors note that this assumes one shared response delay across people. Our method replaces that shared-delay assumption with a fitted per-subject delay, and it decodes single subjects rather than cohort averages.
Two papers supply competing methods. Subject Disentanglement Neural Network for Speech Envelope Reconstruction from EEG (SDN-Net) removes subject identity from a deep envelope decoder. It does this by minimizing mutual information with a subject classifier that identifies people from EEG at 99.8% accuracy, and it beats domain-adversarial training and Wasserstein-GAN alignment. It decodes only the envelope and needs a large network, and its behavior with few source subjects is untested. MindBridge: A Cross-Subject Brain Decoding Framework handles new subjects in fMRI by re-initializing a shallow subject-specific input layer ("reset-tuning") on as few as 500 samples. That is the unconstrained version of what we propose, and we test it head to head. Relate auditory speech to EEG by shallow-deep attention-based network supplies the benchmark numbers for the match-mismatch task.
One paper constrains the design. The primate recordings in Sensory information in local field potentials and spikes from visual and auditory cortices: time scales and frequency bands show that different frequency bands carry largely independent stimulus information (redundancy under 10%). A single latency shared across bands could therefore be wrong if bands come from different generators with different delays. We treat this as a live risk and test it with an unconstrained per-band ablation.
Finally, the sample-size literature, including Scaling laws for language encoding models in fMRI with three densely scanned subjects, warns that effects estimated from few subjects are inflated and unstable. For this reason we report learning curves with resampling variance rather than single-split numbers.
Research Program
We will build a shared preprocessing pipeline. It filters 64-channel EEG into canonical bands from delta to low gamma, computes analytic signals per band, and produces three feature streams: envelope only, raw phase (fine structure), and delay-aligned phase. On these streams we train a fixed decoder family at matched capacity: a linear backward model and a small convolutional network. The tasks are speech-envelope reconstruction and match-mismatch classification.
The central measurement is a subject-count learning curve. For each training-set size $N_{train}$ from 2 up to the cohort maximum, we draw many random subject subsets. We train on each subset and test on disjoint held-out subjects who contribute only a fixed calibration budget, for example 30, 60 and 180 seconds of listening, which is used solely to fit $\tau$ and $\phi$. We record the mean and spread of held-out performance. From these curves we estimate the number of training subjects each feature stream needs to reach a fixed fraction of its within-subject ceiling.
Alongside the curves we run a diagnostic. We fit unconstrained per-band phase offsets and ask whether they fall on a line in frequency. We also check whether the fitted latencies are stable across two independent calibration segments from the same person, which amounts to asking whether latency behaves as a trait. We also measure how well a subject classifier can identify people from each feature stream. If alignment works as claimed, identity decodability should drop sharply for phase and barely change for envelope.
Baselines
The naive reference point is the same decoder trained on concatenated raw envelope and phase features pooled across training subjects, with no per-subject correction. This mirrors the setup that gave near-random cross-subject accuracy in Functional Connectivity and Hilbert-Based Features for Covert Speech EEG Variability Analysis and Classification. Its companion is the envelope-only decoder, which is the default choice in the field. We must beat it at every $N_{train}$ for the phase channel to be worth keeping.
On the ICASSP 2023 match-mismatch task, the file reports an official challenge baseline of 77% accuracy and 78.94% for the full shallow-deep attention network on the blind test set (Relate auditory speech to EEG by shallow-deep attention-based network). Those figures come from the challenge protocol, not from few-subject training. We will therefore report our method on the official split for comparability and also retrain that architecture on our subject subsets. Beating it means higher held-out-subject accuracy at small $N_{train}$ under the same training data, epochs and hyperparameter search budget.
The strongest competitor for cross-subject envelope reconstruction is SDN-Net (Subject Disentanglement Neural Network for Speech Envelope Reconstruction from EEG). The file gives no exact correlation value for it; it states only that it exceeds domain-adversarial and Wasserstein baselines (no reported numeric result found on Lacuna for SDN-Net Pearson correlation). We will reimplement it together with domain-adversarial training, give both the same subject subsets and tuning budget, and let them use the calibration audio of test subjects where their design permits. Beating SDN-Net means matching or exceeding its held-out Pearson correlation at $N_{train} \le 8$ with at least an order of magnitude fewer trainable parameters. We would also claim a win if our aligned features improve SDN-Net itself when fed as its input.
The key ablations isolate the ingredient. The first is an unconstrained per-band complex gain, which tests the linear-in-frequency latency constraint. The second is a MindBridge-style subject-specific linear input layer fit on the same calibration seconds (MindBridge: A Cross-Subject Brain Decoding Framework), which tests constrained against free adaptation. The third applies alignment to the envelope stream, where it should do nothing and so serves as a placebo.
Why not just…?
Why not just drop the phase and use the envelope? Envelope features are more robust, and if aligned phase adds nothing on held-out subjects, this proposal fails. The acoustic evidence says phase carries complementary information: combined AUC 0.93 versus envelope-only 0.83 in Temporal envelope and fine structure cues for dysarthric speech detection using CNNs. Whether this holds for EEG is exactly what the envelope-only curve against the aligned-combined curve will settle.
Why not just fine-tune a subject-specific input layer on the calibration data? This is the real threat, since reset-tuning in MindBridge: A Cross-Subject Brain Decoding Framework rescues new subjects from 500 samples. Our latency correction is a special case of such a layer with far fewer parameters. The honest prediction is that it wins at small calibration budgets, 30 to 60 seconds, and loses or ties at larger ones. The calibration-budget sweep will locate that crossover, and if free fine-tuning dominates everywhere we will report the method as unnecessary.
Why not just use adversarial subject disentanglement? SDN-Net already removes identity from envelope decoders (Subject Disentanglement Neural Network for Speech Envelope Reconstruction from EEG). However, it learns what to remove from the training subjects themselves, and that learning becomes unreliable when there are only a handful of them. Its few-source-subject behavior is unreported. The learning curves compare the two directly, and the combined run tests whether the approaches are complementary.
Why not just average across subjects? Cohort alignment doubles delta tracking in Neural Entrainment to Natural Speech Envelope Based on Subject Aligned EEG Signals, but it requires a reference cohort at inference and cannot decode one listener. We include it as an upper reference. We also test whether inserting our per-subject delay before averaging raises the averaged correlation further, which would directly show that the shared-delay assumption was costing accuracy.
Proposed Experiments
The primary datasets are the ICASSP 2023 Auditory EEG Challenge data for match-mismatch accuracy and the Auditory EEG Decoding Dataset for envelope reconstruction measured by Pearson correlation, both 64-channel continuous-speech listening. If it can be obtained, we will also use the 22-subject Mandarin story cohort, whose time-reversed-speech control lets us check that aligned phase tracks intelligible speech specifically. As a stress test we will use the Overt/Covert speech dataset. It has no continuous stimulus, so alignment must use cue-locked, class-agnostic averages. We expect weaker gains there relative to the pooled 59.14% ± 4.32%, and weak gains would mark the method's boundary.
The decisive checks are three. First, fitted per-band phase offsets should be well explained by a line in frequency, with test-retest-stable latencies. If they scatter, the mechanism is wrong and the unconstrained ablation should match or beat the constrained one. Second, aligned phase should close most of the within-to-cross-subject gap while envelope alignment does nothing. Third, the aligned combined decoder should reach a fixed held-out accuracy with fewer training subjects than every baseline, with non-overlapping resampling intervals at $N_{train} \le 8$. We will report subset-level variance throughout, so that any gain is shown not to be a small-sample fluke of the kind the winner's-curse literature warns about.
References
- Functional Connectivity and Hilbert-Based Features for Covert Speech EEG Variability Analysis and Classification
- Temporal Envelope and Fine Structure Cues for Dysarthric Speech Detection Using CNNs
- Neural Entrainment to Natural Speech Envelope Based on Subject Aligned EEG Signals
- Subject Disentanglement Neural Network for Speech Envelope Reconstruction from EEG
- MindBridge: A Cross-Subject Brain Decoding Framework
- Relate Auditory Speech To Eeg By Shallow-Deep Attention-Based Network
- Sensory information in local field potentials and spikes from visual and auditory cortices: time scales and frequency bands
- Scaling laws for language encoding models in fMRI