Brain-imaging foundation models have to decide how to cut a 4D fMRI scan into pieces a transformer can read. The current options each throw away something important. Averaging into a few hundred atlas regions discards almost all spatial detail. Projecting the cortex onto a 2D "flat map" discards the subcortex and depth. Slicing the raw volume into cubes creates tokens that straddle sulcal banks and mix gray matter with white matter and cerebrospinal fluid. Separately, the structural wiring of the brain, measured by diffusion MRI tractography (a method that traces bundles of nerve fibers through white matter), is known to shape functional activity, yet the large fMRI models ignore it. We propose to fix both problems with one design. Tokens are small, anatomically coherent 3D pieces of gray matter, both cortical and subcortical, each carrying its internal voxel pattern rather than an average. Attention between any two tokens is biased by how strongly they are linked by white-matter tracts.
The motivation is a specific empirical tension. In Scaling Vision Transformers for Functional MRI with Flat Maps, flat-map models reach 98.9% on HCP-YA Task21 state decoding but give a "null result" on clinical traits against simple functional connectivity. Dense volume models do best on HCP-A age (53.4%), which the authors attribute to retained morphometry. Our bet is that the volume model wins for partly the right reasons, keeping subcortex and 3D context, and partly the wrong ones, spending capacity on non-neural voxels with no anatomical prior. A gray-matter-only, tract-aware token set should keep the first advantage and remove the second.
Core Claim
Tokenizing fMRI into tissue-coherent gray-matter supervoxels, with a learnable attention bias derived from tractography, will do three things together. It will match flat maps on state decoding. It will beat both flat maps and cubic-patch volume models on subject-trait prediction. It will beat static functional connectivity on clinical traits, which no model in the Lacuna file has done. The claim is falsifiable and deliberately narrow. The contribution is attributable to structural connectivity only if a tract bias beats a bias built from spatial distance alone and a bias built from degree-preserving shuffled tracts. It is attributable to volumetric context only if it survives a late-fusion baseline that simply appends T1 morphometry features.
Relation to Prior Work
Three works come closest, and none does this. Scaling Vision Transformers for Functional MRI with Flat Maps compares flat-map, parcellation and volume inputs but uses no structural wiring. Its positional embeddings organize along functional networks, and its flat maps cannot represent the hippocampus, thalamus or striatum. We reuse its masked-autoencoder pretraining recipe and its BRAINMARKS evaluation, and we change the token geometry and the attention prior. Joint Graph Convolution for Analyzing Brain Structural and Functional Connectome learns region-specific coupling between structural and functional graphs and finds that coupling varies across cortex. It does this on 80 regions whose time series are averages. We ask whether that coupling survives, and helps, at the scale of a few thousand tokens that keep their internal voxel patterns. MMGPL: Multimodal Medical Data Analysis with Graph Prompt Learning compensates for 2D patching of 3D scans with graph prompts whose edges are GPT-4 concept links. Its graph is semantic rather than anatomical, and its backbone stays 2D-pretrained, a limitation its authors acknowledge.
The method borrows from classical structure-function work. The idea of down-weighting links that lack a physical highway comes from capacity-weighted sparse graphical models (A Novel Sparse Graphical Approach for Multimodal Brain Connectivity Inference). Smoothing tract counts with a heat kernel, so that indirect paths count and hubs do not dominate, comes from Brain Functional Connectivity Estimation Utilizing Diffusion Kernels on a Structural Connectivity Graph.
Two papers constrain the design. Connectome-constrained Latent Variable Model of Whole-Brain Neural Activity uses a hard connectome mask, which works for the complete C. elegans wiring diagram. Macroscale tractography has many false negatives, so we use a soft, per-head learnable bias instead of a mask. Domain-Informed Spline Interpolation and Bayesian Estimation of Probabilistic Atlas for Anatomically-Informed Functional MRI Group Analyses show that shift-invariant resampling and tissue-agnostic pipelines bleed BOLD across sulci and into white matter. This is the argument for restricting both resampling and tokens to gray matter. Both methods are too expensive to run inside a deep model, so we apply their principle once, at tokenization, instead of their machinery.
Research Program
Tokenizer. We build the tokenizer from each subject's T1 image and cortical surfaces. Cortical gray-matter voxels are grouped into supervoxels of roughly 30–60 voxels at 2 mm. The grouping is constrained so that no token contains voxels whose nearest pial-surface points lie on opposite sulcal banks. Subcortical nuclei are split into supervoxels that stay within each nucleus. This yields roughly two to three thousand tokens per frame. BOLD is resampled only within the gray-matter domain, a cheap tissue-masked approximation to domain-informed interpolation. Each token is embedded by a small point-set encoder that sees its voxel values over a short temporal window together with their coordinates. The within-token pattern is therefore preserved rather than averaged away.
Tract bias. From each subject's diffusion MRI we assign tractography streamline endpoints to tokens. This gives a count matrix $S$, which we row-normalize and smooth with the heat kernel $K_\tau = \exp(-\tau L)$, where $L$ is the graph Laplacian of $S$ and $\tau$ sets how far indirect paths reach. Attention logits in head $h$ become
$$ a^{h}_{ij} = \frac{q^{h}_i \cdot k^{h}_j}{\sqrt{d}} + \lambda_h \log\big(K_\tau(i,j) + \epsilon\big), $$
where $q^h_i$ and $k^h_j$ are the query and key vectors of tokens $i$ and $j$, $d$ is the head dimension, $\epsilon$ is a small floor, and $\lambda_h$ is a learned scalar. Because $\lambda_h$ is learned per head, some heads can ignore anatomy entirely while others follow tracts. Local within-hemisphere neighbors are always included.
For datasets without subject diffusion MRI, such as most of ABIDE, we warp a group template matrix built from HCP-YA onto the subject. The gap between template and subject-specific tracts is measured directly on HCP-A, where both are available.
Pretraining and analysis. Pretraining follows CortexMAE's masked reconstruction on HCP-YA resting and task data, with the same masking ratio and token budget. After training, the learned $\lambda_h$ and the per-token tract reliance give a voxel-scale map of structure-function coupling. We compare this map against the frontal and parietal versus sensory gradient reported by Joint-GCN.
Baselines
The naive reference is static functional connectivity: a Pearson correlation matrix over the 400-region parcellation fed to logistic or ridge regression. This is the baseline that flat-map foundation models failed to beat on ADNI and ABIDE (Lane et al.). For it we found no reported result on Lacuna for the exact ADNI or ABIDE scores, so we rerun it on identical splits with nested cross-validation.
The strongest existing representation-learning competitors are CortexMAE with flat-map input and its dense-volume variant. The file reports 98.9% on Task21, 31.0% on NSD COCO24 for the flat-map model, and 53.4% HCP-A age accuracy for the volume model (Lane et al.). We retrain both from the released recipe on our preprocessing, with matched pretraining data, GPU-hours and hyperparameter search budget. Beating them means two things. On Task21 and NSD we must stay within noise of the flat map. On HCP-A age and on ADNI and ABIDE diagnosis we must exceed both, with bootstrap confidence intervals that exclude them.
For multimodal structure-function modeling the reference is Joint-GCN, which reports 84.9% sex accuracy, AUC 0.907, and 1.95-year age MAE on NCANDA (Joint Graph Convolution for Analyzing Brain Structural and Functional Connectome). We rerun it at 80 and at 400 regions on HCP-A and on our other datasets. We also evaluate our model on NCANDA if access permits.
MMGPL's 82.3% on ADNI-3CLS and NeuroH-TGL's 72.19% on ADNI three-way classification (NeuroH-TGL) use different inputs and splits: MMGPL uses PET and structural MRI, while NeuroH-TGL uses AAL-region fMRI. We report them for context but do not treat them as head-to-head.
The ablations isolate each ingredient. The first uses gray-matter tokens with no bias. The second puts the tract bias on ordinary cubic patches. The third replaces tracts with a bias from Euclidean or geodesic distance. The fourth uses a degree- and distance-preserving shuffled tract matrix. The fifth drops subcortical tokens. All ablations share the same budget.
Why not just…?
Why not just append T1 morphometry features to a flat-map or connectivity model? This is the most serious threat. The volume model's age advantage in Lane et al. may be entirely structural, and a vector of cortical thickness and subcortical volumes, fused late with connectivity, could capture it cheaply. Joint-GCN reports that its late-fusion comparison reached 83.1% against 84.9% for joint modeling, which is suggestive but a small margin. We therefore run this late-fusion baseline with full tuning. If it matches our model on trait prediction, we will conclude that the benefit is morphometric rather than structure-functional and drop the central claim.
Why not just use a finer parcellation with SC-plus-FC graph models? Moving Joint-GCN from 80 to 400 or 1000 regions recovers some resolution at low cost. However, it still averages each region's time series, and that averaging is exactly the spatial detail Lane et al. estimate parcellation discards. The ablation that feeds our model region-averaged tokens tests whether within-token voxel patterns matter. If that ablation matches the full model, the finer-parcellation route suffices.
Why not just let attention learn connectivity, or use a distance prior? Structural connectivity is strongly confounded with spatial proximity, so any locality bias might produce the gains. The distance-only and shuffled-tract ablations settle this. We claim a tract effect only if real tracts beat both.
Why not just add subcortical region tokens to the flat map? This hybrid fixes the subcortical blind spot described in Gap 2 without any volumetric tokenizer, and we include it as a baseline. Our model should beat it only if the 3D gray-matter geometry or the tract prior adds information beyond simply including the subcortex.
Proposed Experiments
We pretrain on HCP-YA and evaluate on BRAINMARKS. State decoding uses HCP-YA Task21 and NSD COCO24. Trait prediction uses HCP-A for age and sex, ADNI for diagnosis, ABIDE for autism versus controls, and ADHD-200 and PPMI. Metrics are those of BRAINMARKS: accuracy and balanced accuracy, AUC for diagnosis, and MAE plus binned accuracy for age. Results use five seeds and site-held-out splits for the multi-site sets.
Four checks are decisive. First, the full model must beat both static connectivity and connectivity plus morphometry late fusion on at least ADNI and ABIDE; otherwise the null result stands. Second, real tracts must beat distance-matched shuffled tracts, and subject-specific tracts should beat template tracts on HCP-A. If template tracts are just as good, subject-level structure is not being used. Third, removing subcortical tokens should hurt trait prediction more than state decoding, as the flat-map findings predict. Fourth, the learned per-token tract reliance should correlate with Joint-GCN's regional coupling pattern when aggregated to its 80 regions. If it does, the voxel-scale map is anatomically meaningful rather than an artifact of optimization.
References
- Scaling Vision Transformers for Functional MRI with Flat Maps
- Scaling Vision Transformers for Functional MRI with Flat Maps
- Joint Graph Convolution for Analyzing Brain Structural and Functional Connectome
- MMGPL: Multimodal Medical Data Analysis with Graph Prompt Learning
- A Novel Sparse Graphical Approach for Multimodal Brain Connectivity Inference
- Brain Functional Connectivity Estimation Utilizing Diffusion Kernels on a Structural Connectivity Graph
- Connectome-constrained Latent Variable Model of Whole-Brain Neural Activity
- Domain-Informed Spline Interpolation
- Bayesian Estimation of Probabilistic Atlas for Anatomically-Informed Functional MRI Group Analyses
- NeuroH-TGL: Neuro-Heterogeneity Guided Temporal Graph Learning Strategy for Brain Disease Diagnosis