Electroencephalography (EEG) records the brain's electrical activity through scalp electrodes with millisecond timing but blurry spatial detail. Functional MRI (fMRI) measures blood-oxygen changes with millimeter detail across the whole brain, but those changes lag neural firing by several seconds. Models that predict fMRI volumes from simultaneously recorded EEG, which would give a cheap sensor a window into deep brain activity, have improved quickly. The best of them are still black boxes. They output a full 3D volume without showing which EEG features drove which part of the brain, and their outputs need not respect anatomy. We propose to force the translation through a narrow, readable intermediate layer. Each unit in this layer stands for one region of a standard brain atlas. A digital atlas is a reference brain to which every subject is geometrically warped, a step called spatial normalization. Each unit predicts that region's slow blood-oxygen time course through its own learned delay kernel. A separate, tightly constrained local pathway then adds detail inside regions.
The core claim is that most of the information EEG carries about fMRI passes through region-level signals. If that is true, a model whose coarse, global content is confined to atlas-region units and whose fine, local content is confined to a within-region residual should match an unconstrained transformer in voxel accuracy. It should also give explanations whose spatial location is guaranteed by construction rather than inferred after the fact. A second payoff comes from letting each unit's spatial footprint deviate softly from its atlas boundary. The deviations that recur across subjects then become a measurable estimate of where atlas warping misplaces function. This turns spatial normalization from a preprocessing assumption into something the model tests.
What exists and what is missing
The closest system is NT-ViT. It converts EEG into spectrograms, passes them through a transformer encoder-decoder to synthesize fMRI volumes, and aligns EEG and fMRI latents with a domain-matching loss. It reports roughly a 10× RMSE reduction on the Oddball dataset and a 5× reduction on NODDI relative to prior work. It does not place electrodes or outputs in a standard anatomical space, gives no account of which inputs drive which regions, and relies on global attention without boundary calibration. Our proposal keeps its task, data and leave-one-subject-out protocol. It replaces its free latent with an atlas-indexed one.
Three papers motivate the bottleneck design and also constrain it. Concept Bottleneck Models route predictions through human-named intermediate variables. Coherent Concept-based Explanations shows in 2D dermoscopy that a Dice-style coherence loss, which penalizes concept activations outside the lesion, improves accuracy and keeps explanations on the anatomy. On PH² it reports 96.00% accuracy against 85.33% for the black-box equivalent. We borrow the coherence loss, but our "concepts" are atlas regions in 3D normalized space, not dermoscopic features. Bridging Brains and Concepts is the cautionary result. Its 214-question semantic bottleneck for fMRI decoding costs accuracy, with CLIP similarity of 0.885 versus 0.923 for the unconstrained model on Subject 01, and it loses spatial geometry and fine texture. We take this as evidence that a bottleneck alone will be incomplete. That is why our design includes a within-region residual pathway, constrained so it cannot smuggle region-level information around the bottleneck.
The local-global split follows the lesson of CKD-TransBTS: global transformer features blur boundaries, and a separate local calibration path is needed. Our split is defined anatomically rather than by network depth. The coarse path owns region means and the local path owns zero-mean within-region structure. For electrodes, Uni-NTFM shows that embedding electrode positions and separating time-domain from frequency-domain paths handles heterogeneous montages. We reuse that encoder idea and place electrode coordinates in MNI space so that EEG and fMRI share one coordinate frame. The classical analysis in Building an EEG-fMRI Multi-Modal Brain Graph convolves EEG band power with a fixed hemodynamic response function and correlates it with fMRI components. It supplies our simplest serious baseline, and our per-region learned delay kernel generalizes its fixed kernel. BrainGFM documents that learned representations tied to one parcellation often fail to transfer to another. This motivates the atlas-swap test below.
Research Program
The model has four parts. First, an EEG encoder embeds each electrode by its MNI-projected scalp position and processes each channel through a convolutional time path and a Fourier frequency path, producing features $h(t)$. Second, a bank of $K$ region units, one per parcel of a chosen atlas, maps $h(t)$ to a scalar latent drive $z_k(t)$. Each unit convolves its drive with its own learned delay kernel, so the slow hemodynamic lag is explicit and inspectable. Third, each unit owns a spatial footprint $F_k$ over voxels. The footprint is initialized to its atlas mask and is free to change, but is penalized by a Dice-based coherence term for mass outside a dilated version of the mask. Fourth, a local convolutional decoder produces a residual volume whose mean within every parcel is projected to zero. Region-level variance can therefore enter only through the units.
The predicted volume at time $t$ is
$$\hat{y}(t) = \sum_{k=1}^{K} (g_k * z_k)(t)\, F_k + r(t),$$
where $g_k$ is unit $k$'s delay kernel, $*$ is temporal convolution, and $r(t)$ is the parcel-mean-free residual. We bound $r$ by an energy penalty. We report how much voxel variance each pathway explains, so readers can see how much of the prediction the readable part carries.
Three analyses follow from the trained model. Delay kernels are compared across regions and subjects to test whether hemodynamic lag differs regionally in a consistent way. Footprint deviations from atlas masks are averaged across leave-one-subject-out folds. Deviations that recur across subjects point to systematic atlas misplacement; deviations that do not recur are treated as noise. Interventions that perturb a single $z_k$ test whether the output changes only inside $F_k$, as the design intends, and how much the residual pathway reacts. Large residual reaction would indicate leakage.
Baselines
The naive reference point predicts each test subject's fMRI as the training-set mean volume. Any method must clear it in RMSE, SSIM and per-parcel temporal correlation.
The classical baseline fits ridge regression from hemodynamically convolved EEG band power, in the five bands used in Building an EEG-fMRI Multi-Modal Brain Graph, to parcel-mean fMRI time courses. Predictions are broadcast back to voxels. This baseline is interpretable, spatially normalized and nearly free. It shares the same atlas, preprocessing and folds, with regularization tuned by nested cross-validation.
The strongest existing method is NT-ViT. The file reports only relative gains for it, roughly 10× lower RMSE and 3.14× higher SSIM on Oddball and 5× lower RMSE and 1.23× higher SSIM on NODDI versus earlier work. No absolute RMSE or SSIM value for NT-ViT is reported on Lacuna. We will therefore retrain it on our identically preprocessed, MNI-registered data under the same leave-one-subject-out splits, with a matched parameter count and the same hyperparameter search budget, and report absolute numbers for both models. Beating NT-ViT means matching its voxel RMSE and SSIM within the fold-to-fold standard error while exceeding it on per-parcel correlation and on the faithfulness and stability measures below. We do not claim that interpretability must buy higher raw accuracy.
Three ablations isolate the ingredients. The first removes the zero-parcel-mean projection on the residual, which shows whether region information leaks around the bottleneck. The second fixes footprints to hard atlas masks, which shows whether soft coherence matters. The third replaces per-region delay kernels with one canonical hemodynamic response function.
Why not just…?
Why not just train NT-ViT and explain it afterward with attribution maps? Post-hoc maps show where gradients are large, not which mechanism produced an output, and nothing ties them to atlas regions. We will compute integrated-gradient maps on our retrained NT-ViT and compare them with our intervention-based maps on the same test region. We will also measure how much each changes under a second, independently seeded training run. If post-hoc maps prove as stable and as spatially local as the bottleneck's interventions, the case for building the bottleneck in weakens, and we will say so.
Why not just regress parcel averages linearly from band power? This is the real threat, given that the paired cohorts are small: 17 subjects in Oddball and 15 in NODDI. If ridge regression matches our per-parcel correlation, then the deep encoder adds nothing at the region level. The remaining contribution would be the within-region residual and the footprint analysis. Our main comparison is built to settle exactly this question, and a tie would be reported as a finding about how much nonlinear EEG information reaches fMRI.
Why not just add an unconstrained region-wise readout head to a transformer? Without the zero-mean residual and the coherence penalty, the head's units are free to encode anything, and the region labels become decorative. This is the same leakage that Multimodal Concept Bottleneck Models counters with non-negativity and sparsity constraints. The first ablation measures this directly through the intervention-locality score.
Proposed Experiments
All experiments use the simultaneous EEG-fMRI Oddball and NODDI datasets with leave-one-subject-out evaluation, following NT-ViT. fMRI is registered to MNI152 and electrodes are projected to MNI scalp coordinates. Accuracy is reported as voxel RMSE, SSIM and per-parcel Pearson correlation. Faithfulness is the fraction of output change falling inside $F_k$ when $z_k$ alone is perturbed.
The first decisive check asks whether the bottleneck model reaches NT-ViT's voxel accuracy while keeping intervention locality high. Failure here would repeat the completeness penalty seen in Bridging Brains and Concepts.
The second check is atlas robustness. We train separately with two parcellations of different granularity and test whether region-level explanations agree where parcels overlap. This directly probes the atlas-dependency problem raised by BrainGFM.
The third check is a registration stress test. We inject controlled random deformations into the test subjects' warps and measure whether the learned soft footprints absorb the error better than the fixed-mask ablation. We also test whether footprint deviations learned from real data correlate across subjects above a permutation null. Only that last result would justify treating learned footprints as evidence about where atlas normalization goes wrong.
References
- NT-ViT: Neural Transcoding Vision Transformers for EEG-to-fMRI Synthesis
- Concept Bottleneck Models
- Coherent Concept-based Explanations in Medical Image and Its Application to Skin Lesion Diagnosis
- Bridging Brains and Concepts: Interpretable Visual Decoding from fMRI with Semantic Bottlenecks
- CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer With Modality-Correlated Cross-Attention for Brain Tumor Segmentation
- Uni-NTFM: A Unified Foundation Model for EEG Signal Representation Learning
- Building an EEG-fMRI Multi-Modal Brain Graph: A Concurrent EEG-fMRI Study
- A Brain Graph Foundation Model: Pre-Training and Prompt-Tuning for Any Atlas and Disorder
- Multimodal Concept Bottleneck Models