Volumetric Masking: Removing the Neighbor Shortcut in EEG Pretraining

EEG foundation models learn by hiding pieces of a recording and predicting them back. Scalp EEG, however, is extremely redundant along each axis. Neighboring electrodes see nearly the same signal, because electrical fields spread through skull and scalp, and adjacent time patches of one channel are strongly autocorrelated. A model can therefore fill in a hidden patch by smoothing between its immediate spatial or temporal neighbors, and never learn how activity at one place and time constrains activity somewhere else. We propose to treat this as the EEG version of what medical imaging calls volumetric context loss: models that process a 3D scan slice by slice lose the information that only appears when structures are followed across depth. The fix we propose sits in the training objective, not the architecture. We hide contiguous blocks of the recording defined jointly in physical head space and time, so that every hidden patch must be predicted from context outside its own neighborhood.

The opportunity is concrete. Two open, widely used EEG foundation models (CBraMod, LaBraM) release code and checkpoints. Both pretrain by masked reconstruction with per-patch masking, and neither imposes any cross-axis consistency on what it reconstructs. If per-patch masking really lets these models get away with local interpolation, then a change to the mask alone should yield representations that rely more on long-range spatial-temporal structure. It should also make them degrade more gracefully when electrodes are missing, which is the hardware-heterogeneity problem that rigid 3D EEG grids cannot handle.

Core Claim

Our claim is that, in masked EEG pretraining, out-of-plane context is lost mainly through the objective rather than the architecture. LaBraM already lets every channel-time token attend to every other one. The masking task simply never requires that ability. We will show three things. First, a simple test-time probe can measure the "neighbor shortcut" directly. Second, masking 3D blocks (a geodesic disk of electrodes on the head, held hidden over a span of time) reduces that shortcut. Third, the reduction carries over to downstream gains that are largest on tasks whose signal is a spatially spreading event, and to robustness under sparse montages. If block masking gives no advantage over random masking at the same mask fraction, the claim is false, and we say so up front.

Formally, for a recording with electrodes $c$ at measured 3D scalp positions and time patches $t$, a volumetric mask is centered at an electrode $c_0$ and patch $t_0$:

$$ M(c_0,t_0)=\{(c,t): d_{\text{geo}}(c,c_0)\le r,\ |t-t_0|\le \tau\} $$

Here $d_{\text{geo}}$ is the great-circle distance between electrode positions on a fitted head sphere, $r$ is the spatial radius, and $\tau$ is the temporal half-width. Several such blocks are sampled until a target fraction of tokens is hidden. The mask is defined on electrode coordinates rather than on a fixed $9\times 9$ grid, so it works for any montage and never introduces the zero-padded cells that G3DCT and similar grid tensors require.

Positioning Against Prior Work

The motivating evidence comes from volumetric imaging. Roth et al. showed that slice-by-slice pancreas segmentation produces fragmented organs that miss whole segments across slices. Han et al. found that slice-wise generation tears anatomy apart unless cross-slice consistency is enforced. Most relevant to our probe, Wolf et al. found that vision-language models answer spatial questions about scans at chance, because they lean on memorized priors instead of the out-of-plane pixels. That pattern of a model succeeding while ignoring the evidence it should use is exactly what we want to test for in EEG.

Imaging fixes this mostly through architecture: native 3D networks (Çiçek et al.) and linear-cost global models such as Swin-UMamba. We deliberately do not follow that route. EEG models that already have global attention would gain nothing from a wider receptive field if the training task never rewards using it.

On the EEG side, a number of works build 3D tensors by placing electrodes on a 2D scalp grid and stacking along time or frequency. Examples are ConSwinFormer with a 3D Swin Transformer, G3DCT with 3D residual convolutions on a $9\times 9$ grid, and the 3D-CNN-MSTA-Transformer, which factorizes 3D kernels into 2D-plus-1D for speed. These works supply our notion of a spatial neighborhood. All of them are supervised and grid-bound, and none asks whether the model actually uses joint context. CBraMod splits attention into parallel spatial and temporal branches and pretrains with patch masking on TUEG. LaBraM tokenizes single-channel patches and predicts their Fourier codes. These two supply our architectures and pretraining pipelines; we change only what is hidden. STGATE shows that learned connectivity beats fixed physical adjacency. This constrains our design: geodesic blocks should be compared against blocks defined by learned connectivity, not assumed to be optimal.

UniEFS is the closest work that explicitly links EEG to volumetric context. It maps EEG into an fMRI latent space pretrained with volumetric masking on HCP. Its masking is applied to the fMRI side, and the authors note that the EEG input is not structured spatially. Our work addresses the EEG encoder itself and asks no cross-modal question.

Research Program

The first component is a measurement: a neighbor-shortcut index for any masked EEG model. For a hidden patch, we compare reconstruction error in two conditions. In one, all other tokens are visible. In the other, we additionally hide the patch's geodesic and temporal neighbors and leave only distant context. A large relative rise in error means the model relies on local interpolation. The index is computed on held-out TUEG data and swept over neighborhood radius. This gives a curve showing how far each model "looks." We apply it first to the released CBraMod and LaBraM checkpoints, before any training of our own. If those models already reconstruct well from distant context, the premise fails and the project stops there, at the cost of a few GPU-days.

The second component is pretraining. We reproduce CBraMod pretraining on TUEG with its released code, and a smaller LaBraM-Base run with its code. For each, we train three masking variants under identical architecture, data, steps, and masked-token fraction: the original random patch masking, random masking at a raised fraction, and volumetric geodesic-block masking. We sweep $r$ and $\tau$ on a small validation budget. A frequency-aware variant also hides one band of the reconstruction target inside the block. This tests whether cross-frequency coupling, of the kind Van Dyck et al. report between lower bands and low gamma, can be learned as context.

The third component is evaluation along three axes: standard downstream fine-tuning, robustness to missing channels, and a check of whether gains line up with the shortcut index.

Baselines

The naive reference point is a model with no pretraining, trained from scratch on each downstream task with the same architecture. For motor imagery we also include classical filter-bank spatial filtering. FBCSP reports a mean kappa of 0.569 on BCICIV-2a, and any pretrained model that cannot clear it on that dataset has no case.

The strongest existing methods are the released foundation models. On TUEV, LaBraM-Huge reports balanced accuracy 0.6616 and on TUAB an AUROC of 0.9162 (LaBraM). CBraMod reports Cohen's kappa 0.5041 on FACED and balanced accuracy 64.17% on PhysioNet-MI (CBraMod). We found no reported CBraMod or LaBraM accuracy on CHB-MIT in the file, only FLOPs, so we will establish that number ourselves.

Our primary comparison does not set our model against these published figures, because they reflect different compute. Instead we compare our re-run of each model's own recipe with that recipe plus volumetric masking. Data, steps, hidden-token fraction, fine-tuning protocol, and hyperparameter search budget will all be identical, with three seeds per configuration. Beating the baseline means a gain larger than seed variance on at least TUEV and TUAB. It also means at least matching the published numbers when we use the released checkpoint as initialization and continue pretraining with volumetric masks for a fixed small budget.

The ablation that isolates the key ingredient is random masking at a raised fraction, chosen so that the expected number of visible within-neighborhood tokens per hidden patch equals that under block masking. If this matches block masking, the benefit came from "more hiding," not from contiguous volumetric structure. A second ablation shapes blocks by learned functional connectivity, in the style of STGATE, instead of by geodesic distance. This tells us whether physical scalp geometry is the right definition of neighborhood.

Why not just…?

Why not just raise the random mask ratio? Hiding more patches at random also starves the model of neighbors, and it is the most serious alternative. The difference is in what it leaves visible. At a moderate ratio, random masking almost always leaves some adjacent electrode or time step for a hidden patch. At very high ratios it leaves almost nothing anywhere, and reconstruction becomes guessing. Block masking removes local context while keeping distant context dense. The matched-neighbor ablation above settles this directly. If raised-ratio random masking ties block masking on both the shortcut index and downstream results, we will report that the simpler recipe suffices and drop the volumetric framing.

Why not just use a full 3D architecture, as imaging does? In imaging, the slice-wise model literally cannot see across slices, so architecture is the bottleneck (Roth et al.). LaBraM already attends jointly across all channel-time tokens, yet nothing in its objective requires distant context. Our shortcut probe on the released LaBraM checkpoint tests whether a joint architecture alone avoids the shortcut. If LaBraM's index is low while CBraMod's split-attention model is high, architecture matters after all, and we will report the interaction between masking and architecture rather than a masking-only story.

Why not just feed a 3D scalp-grid tensor to a 3D CNN, as in G3DCT or ConSwinFormer? These models gain spatial locality as a built-in assumption, but they are supervised. They require fixed grids with padded empty cells, and ConSwinFormer's 83.99% on BCICIV-2a (ConSwinFormer) comes with wide variation across subjects. A local-convolution prior also encourages exactly the neighbor interpolation we are trying to suppress. We include a grid-CNN front end on BCICIV-2a and SEED as a supervised comparison. We expect it to be competitive on clean trial data but to fail the missing-channel test, since its input shape is fixed.

Proposed Experiments

The first decisive check is the shortcut index on the released checkpoints, before any training of our own. The premise requires that reconstruction error rises steeply once geodesic neighbors within a few centimeters are hidden. Next comes the matched pretraining comparison: three masking variants times two backbones times three seeds, all on TUEG. These models are then fine-tuned on TUEV, TUAB, CHB-MIT, FACED, PhysioNet-MI, and BCICIV-2a with each backbone's published protocol.

We have a prediction for where gains should appear. They should concentrate where the label depends on how a pattern is laid out and spreads across the scalp: TUEV's spike and sharp-wave events, which have spatial fields, and CHB-MIT seizure onsets, which propagate. They should be smallest on tasks dominated by band power at a few sites. If gains are uniform across tasks, the improvement is generic regularization and not recovery of out-of-plane context.

The robustness test fine-tunes on the full montage and then evaluates on TUAB and TUEV with electrodes removed down to sparse subsets. These range from standard reduced sets to a four-channel, headband-like configuration, reflecting the heterogeneity LaBraM emphasizes. Volumetric masking should lose less accuracy as channels are removed, because pretraining taught it to infer whole missing regions.

Finally, across all checkpoints, we test whether a lower shortcut index predicts larger downstream gains. A positive correlation would establish the index as a cheap pretraining diagnostic. No correlation would mean that whatever block masking does, it does not work through the mechanism we propose.