Decoders that read imagined speech from brain recordings have a calibration problem. A new user must produce many labeled imagined trials, and these trials are the worst possible training data. They carry no audio or muscle movement, so nobody can check when the user actually "said" the word or whether they said it at all. Spoken trials are the opposite: their onsets are verifiable, their signal is stronger, and they are fast to collect while a patient can still talk, for example early in ALS. The obvious move is to train on spoken trials and test on imagined ones. Published work says this fails. On EEG, models trained on articulated speech transfer poorly to imagined speech, and pooling the two modes does not significantly help (Using articulated speech EEG signals for imagined speech decoding).
We propose to make that transfer an explicit training objective. Across a group of previous users, we meta-learn (train the starting point of a model so that a few gradient steps on a new task work well) a shared network together with a small low-rank adapter, a cheap add-on that changes a frozen network through a product of two thin matrices. The training rule is unusual in one way. The adapter is fit to a user's spoken trials in the inner loop and scored on that same user's imagined trials in the outer loop. The shared network therefore learns which directions of change, estimated from spoken data, also hold for imagined speech, and which ones only describe muscle activity and auditory feedback. At deployment, a new user records a few minutes of spoken words, the adapter is fit to those trials, and the decoder is used for imagined speech, with optionally a handful of imagined trials added.
Core Claim
Spoken-to-imagined transfer fails because ordinary training absorbs whatever separates the two modes into the network's main weights. Nothing in that training forces the spoken-specific part to be separable. Block et al. make the structural version of this point for low-rank adaptation (Provable Meta-Learning with Low-Rank Adaptations). Standard retraining on many tasks bakes the average of their low-rank updates into the base weights, and that average is generally high-rank, so a later low-rank adapter cannot undo it. Their meta-learned alternative, LoRA-ML, trains the base so that per-task low-rank updates do the task-specific work. We keep that mechanism and change what the inner and outer loops see.
Write $\theta$ for the shared network weights, $A$ for a frozen random orthogonal down-projection, and $B_0$ for a meta-learned initial up-projection. For each source subject $s$, $\mathcal{L}^{\text{spk}}_s$ is the loss on a few of that subject's spoken trials and $\mathcal{L}^{\text{img}}_s$ is the loss on that subject's imagined trials. The objective is
$$\min_{\theta,\,B_0}\ \sum_s \mathcal{L}^{\text{img}}_s\!\big(\theta,\ B_0 - \alpha \nabla_{B}\mathcal{L}^{\text{spk}}_s(\theta, B_0)\big),$$
where $\alpha$ is the inner-loop step size, and in practice there are a few inner steps. Only $B$ adapts. Zhu et al. show that a frozen random $A$ with a trained $B$ matches standard low-rank adaptation and generalizes better out of domain (Asymmetry in Low-Rank Adapters of Foundation Models). Restricting the inner loop to a small space also follows the lesson of Meta-Learning with Latent Embedding Optimization: gradient steps in a large weight space overfit when the support set is tiny. A small adapting space also keeps second-order meta-gradients affordable.
The new part is the asymmetric support/query split across speech modes. LoRA-ML, Meta-Adapters (Meta-Adapters: Parameter Efficient Few-shot Fine-tuning through Meta-Learning) and the EEG meta-learner MUPS-EEG (Meta Learn on Constrained Transfer Learning for Low Resource Cross Subject EEG Classification) all draw support and query from the same task distribution. None of them is applied to imagined speech. None of them asks an adaptation computed from one behavior to generalize to another.
Research Program
The work has three parts.
Backbone and data. We build a compact temporal-convolution-plus-attention encoder for multichannel recordings, with low-rank adapters on its attention and feed-forward projections. The main testbed is the Delft Articulated and Imagined Speech dataset (DAIS): 62-channel EEG from 20 subjects, each producing the same 5 vowels and 10 words both aloud and in imagination. Because the same subjects do both modes, cross-mode, cross-subject meta-learning is possible under leave-one-subject-out evaluation. The second testbed is VocalMind (VocalMind), 140-channel stereo-EEG with vocalized, mimed and imagined Mandarin words and sentences. The file does not give its participant count. If it is a single participant, we form meta-tasks from held-out word subsets instead of held-out subjects. The question there becomes whether spoken examples of new words calibrate the decoder for imagining those words. Mimed speech then offers an intermediate mode, and we test it as support on its own.
Variants of the training objective. We compare spoken support with imagined query (the proposal), imagined support with imagined query (plain LoRA-ML on imagined data), and a mixed support set at matched recording time. One ablation adds a shared "mode-shift" adapter, fixed across subjects, alongside the per-subject adapter. If a fixed shared offset captures most of the gain, then per-subject cross-mode adaptation is unnecessary and the simpler model wins.
Analysis of what the adapter learns. We measure how much of the adapted update's input-side energy falls on temporal and facial-muscle channels and on high-frequency bands, where spoken-speech EEG carries electromyographic contamination. We compare this with an adapter fit by ordinary fine-tuning. The hypothesis predicts that the meta-learned base makes spoken-derived updates ignore muscle-dominated directions. The ECoG finding that motor-cortex high-gamma activity is shared between spoken and imagined speech, while auditory-cortex gamma is absent during imagination (Imagined speech event detection from electrocorticography and its transfer between speech modes and subjects), predicts the same pattern on VocalMind's depth electrodes: the adapter should keep motor contacts and discard auditory ones.
Baselines
The naive reference point is the DAIS result itself. Imagined vowel classification reached 24.8% from imagined data alone and up to 27% with pre-speech articulated data, against a 20% chance level, and the difference was not significant (Using articulated speech EEG signals for imagined speech decoding). We reproduce it with our backbone under the same subject split so that every later comparison shares one evaluation. A method that does not clearly exceed it is not interesting.
The strongest existing method is plain LoRA-ML applied within the imagined mode (Provable Meta-Learning with Low-Rank Adaptations). It uses the same backbone, adapter rank and meta-training subjects, with imagined support and imagined query. For its most comparable setting, ConvAI2 personalization, it reports 47.4% accuracy against 35.0% for standard retraining. There is no reported result found on Lacuna for LoRA-ML on any neural dataset. This baseline is strongest because it isolates our one change: where the support trials come from. We run it at equal calibration recording time and at equal trial count, since spoken trials can be quicker to segment.
The second strong competitor is multi-subject pretraining by ordinary empirical risk minimization on pooled spoken and imagined data, followed by fine-tuning a standard zero-initialized adapter on the new subject. This is the recipe implicit in BrainECHO's adaptation stage (BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generation). It is also the pretraining that lifted cross-subject MEG accuracy from 0.329 to 0.478 balanced top-10 in LibriBrain100. We also run an EVA-initialized adapter (One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation), with its activation decomposition computed on the new subject's calibration trials, and normalization-only test-time entropy minimization (A Robust Multi-Scale Framework with Test-Time Adaptation for sEEG-Based Speech Decoding).
Every baseline receives the same hyperparameter search budget on source-subject validation folds and the same adapter parameter count where it applies. The ablation that isolates our key ingredient is the swap of spoken support for imagined support with everything else fixed. The proposal counts as beating the field only under two conditions. First, at matched calibration time, spoken-support meta-learning beats imagined-support LoRA-ML on held-out-subject imagined accuracy, with a paired test across the 20 DAIS subjects. Second, it beats ordinary pretraining plus fine-tuning by more than the subject-level standard deviation.
Why not just…?
Why not just pool spoken and imagined data and pretrain on everything? DAIS already reports that pooling gives no significant gain. The Chisco work documents an "imagination penalty": reading-based EEG follows different and more robust dynamics than self-initiated imagined speech (Chisco: An EEG-based BCI dataset for decoding of imagined speech). The rank-mismatch argument explains why pooling should fail with low-rank adapters in particular. We do not rely on that argument alone. Pooled pretraining is a full baseline, and if it matches us the claim fails.
Why not just meta-learn on imagined trials directly? This is the real threat. If imagined trials are as informative per minute as spoken ones, cross-mode support gains nothing. Spoken support might win because of verifiable onsets and higher signal, or it might lose because of muscle contamination. The matched-time comparison settles the question. We also report the curve over mixtures of spoken and imagined support, because the honest result may be that a few imagined trials plus many spoken ones are optimal. For users who can no longer speak but can still attempt or mime speech, VocalMind's mimed mode tests whether the benefit survives without phonation.
Why not just initialize adapters from the new user's activations (EVA) or adapt normalization layers at test time? EVA finds directions of high variance. With a few noisy EEG trials those directions are likely dominated by blinks and muscle activity, which is exactly what spoken-to-imagined transfer must ignore. EVA has no signal telling it which variance matters for imagined speech. Normalization-only adaptation cannot rotate features. The sEEG test-time adaptation study needed an added multi-scale architecture and self-distillation to make it stable, and its single components sometimes hurt individual subjects. Both are cheap enough to run as baselines. They also combine with ours, so we report entropy-based test-time updates on the meta-learned adapter as an add-on rather than a rival.
Proposed Experiments
The primary experiment is leave-one-subject-out on DAIS: 19 meta-training subjects and one held-out subject, rotated. We report imagined vowel and word accuracy against calibration minutes, with support sizes from one to ten spoken trials per class, and add Cohen's kappa for comparability with Chisco-style reporting. On VocalMind we report imagined word accuracy and spectrogram Pearson correlation for held-out words, calibrated from vocalized and from mimed support. There is no reported result found on Lacuna for VocalMind word accuracy, so our own adapter-free decoder serves as the reference.
The decisive checks are three. First, the support-swap ablation must favor spoken support at matched time on DAIS. If it does not, the paper reports that cross-mode calibration does not pay on EEG and turns to the sEEG case. Second, the muscle-channel analysis must show that meta-learned updates place less input energy on electromyography-prone channels and bands than fine-tuned updates do. Without this, any gain could come from an artifact correlated with word identity. Third, the shared mode-shift adapter alone must underperform per-subject cross-mode adaptation. Otherwise the simpler fixed-offset model is the real contribution, and we will say so.
References
- Using articulated speech EEG signals for imagined speech decoding
- Provable Meta-Learning with Low-Rank Adaptations
- Asymmetry in Low-Rank Adapters of Foundation Models
- Meta-Learning with Latent Embedding Optimization
- Meta-Adapters: Parameter Efficient Few-shot Fine-tuning through Meta-Learning
- Meta Learn on Constrained Transfer Learning for Low Resource Cross Subject EEG Classification
- VocalMind: A Stereotactic EEG Dataset for Vocalized, Mimed, and Imagined Speech in Tonal Language
- Imagined speech event detection from electrocorticography and its transfer between speech modes and subjects
- BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generation
- LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scale
- One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
- A Robust Multi-Scale Framework with Test-Time Adaptation for sEEG-Based Speech Decoding
- Chisco: An EEG-based BCI dataset for decoding of imagined speech