To understand the human brain, cognitive neuroscientists and clinicians rely on an array of recording technologies. However, each instrument offers a fundamentally fragmented view. Electroencephalography (EEG) and magnetoencephalography (MEG) capture neural dynamics at the millisecond scale but suffer from poor spatial resolution. Conversely, functional Magnetic Resonance Imaging (fMRI) maps blood-oxygenation changes with millimeter precision but operates on a sluggish delay of several seconds. Structural MRI provides anatomical constraints, while genomics offers the underlying biological blueprint.
Historically, computational models have been siloed within a single modality. Recent breakthroughs have seen the rise of large-scale foundation models optimized for isolated data streams. For instance, Wei-Bang Jiang et al. developed Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI to scale Transformer architectures over thousands of hours of EEG data. Similarly, Daniel Levine et al. introduced BrainLM: A foundation model for brain activity recordings to learn representations exclusively from fMRI volumes.
Yet, as the authors of both studies note, a single-modality focus inherently limits a model's capacity to capture holistic neural states. The frontier of computational neuroscience lies in cross-modal brain modeling: unifying these heterogeneous data streams into a single, cohesive representation space.
Tokenization as a Common Currency
The primary obstacle to cross-modal integration is structural incompatibility. How can a single neural network ingest a one-dimensional electrical time-series (EEG), a three-dimensional metabolic volume (fMRI), and a sequence of text or images simultaneously?
The solution borrowed from modern deep learning is discrete tokenization. By converting continuous, highly disparate signals into a standardized sequence of discrete tokens, transformers can process different modalities through a shared architecture.
[Continuous EEG/MEG] ──> [Sensor Encoder + BrainTokenizer] ──┐
│ ──> [Shared Transformer Core]
[Continuous fMRI] ──> [Discrete Diffusion Tokenizer] ──┘
In BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals, Qinfan Xiao et al. demonstrated this by unifying EEG and MEG. They utilized a physics-aware Sensor Encoder alongside a specialized BrainTokenizer to map distinct electrophysiological signals into a unified latent space, allowing the model to generalize to entirely unseen recording devices.
This tokenization strategy extends beyond combining different neural recordings; it can also bridge brain activity with external sensory stimuli. In Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion, Yizhuo Lu et al. converted continuous fMRI signals into discrete tokens within a unified framework that models vision and language simultaneously. This discrete alignment enables a synergy where joint modeling of images and text directly improves the decoding of brain states.
Preserving Spatial and Geometric Topology
When flattening brain signals into tokens, models risk losing the physical reality of the brain's architecture—its spatial geometry and functional networks. Electrophysiological sensors sit on a curved scalp or cortex, and fMRI voxels belong to complex anatomical regions. Successful cross-modal models must bake this topology directly into their architectures.
To prevent transformers from treating EEG channels as an ordered sequence, Wenchao Yang et al. introduced a "Brain Topology Hierarchy" in THD-BAR: Topology Hierarchical Derived Brain Autoregressive Modeling for EEG Generic Representations. By redefining autoregressive prediction across spatial scales rather than just forward in time, the model explicitly accounts for the brain's physical layout.
This multi-scale requirement is echoed in CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding by Yuchen Zhou et al., which decouples local, rapid electrical bursts from large-scale, slow rhythms to capture the full spatiotemporal spectrum.
Spatial Scale (Anatomy)
▲
│ ┌────────────────────────┐
│ │ Global Brain Networks │ <── Graph Models (BrainGFM)
│ └────────────────────────┘
│ ▲
│ │
│ ┌────────────────────────┐
│ │ Local Cortical Patches │ <── Topology-Aware Transformers (THD-BAR)
│ └────────────────────────┘
└──────────────────────────────────► Temporal Scale (Dynamics)
For volumetric and network-level data, graphs offer a more natural language than grids. In A Brain Graph Foundation Model: Pre-Training and Prompt-Tuning for Any Atlas and Disorder, Xinxu Wei et al. bypassed the limitations of static brain atlases by building a graph-based foundation model. Their framework uses language-guided prompt tuning to align functional connectivity graphs across entirely different clinical disorders and imaging protocols.
On the software side, mapping these complex spatial properties onto viewable surfaces remains critical. James Gao developed Pycortex: an interactive surface visualizer for fMRI to handle the pixel-based mapping of volumetric data onto cortical surfaces—a visualization challenge that researchers are actively trying to adapt to non-volumetric modalities like EEG and electrocorticography (ECoG).
Mathematical Fusion of Disparate Modalities
While deep learning foundation models excel when data is abundant, finding correlations between highly heterogeneous datasets—such as imaging and genomics—often requires rigorous statistical fusion techniques to handle extreme differences in scale and noise.
In Learning Schizophrenia Imaging Genetics Data Via Multiple Kernel Canonical Correlation Analysis, Owen Richfield et al. tackled the challenge of pairing DNA methylation data with fMRI scans. By utilizing Multiple Kernel Canonical Correlation Analysis (MKCCA), the authors successfully mapped non-linear relationships across these distinct biological layers, revealing that the combination of epigenetic and functional imaging data provides a much stronger diagnostic signal for schizophrenia than either modality alone.
Furthermore, interactions between different imaging modalities are rarely strictly one-to-one; a single structural pathway might influence multiple functional networks. To capture these complex relationships, Chunying Jia et al. developed the Consecutive Independence and Correlation Transform (C-ICT) in Consecutive Independence and Correlation Transform for Multimodal Data Fusion: Discovery of One-to-Many Associations in Structural and Functional Imaging Data. This method allows researchers to flexibly fuse diffusion MRI (dMRI), structural MRI (sMRI), and fMRI, exposing "one-to-many" associations that traditional linear fusion techniques miss.
Why Cross-Modal Modeling Works
The fundamental reason cross-modal integration succeeds is that different modalities act as regularizers for one another.
When a model is trained on a single modality, it easily overfits to the idiosyncratic noise, artifacts, and biases of that specific sensor type (such as muscle movement artifacts in EEG or head motion in fMRI). Forcing a model to find a shared representation across multiple modalities compels it to discard sensor-specific noise and retain the true underlying neural generation source.
This principle manifests clearly across several domains:
- Clinical Diagnostics: In Enhanced brain parcellation via abnormality inpainting for neuroimage-based consciousness evaluation of hydrocephalus patients by lumbar drainage, Di Zang et al. demonstrated that combining structural "inpainting" (restoring deformed anatomy in scans) with functional connectivity data allows for far more accurate predictions of consciousness recovery than relying on functional data alone.
- Cognitive Encoding: In In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain, Ethan Hwang et al. built a whole-brain transformer encoder to map cortical responses to complex images. They noted that moving beyond simple visual mapping to capture the brain's true integrative nature requires multi-modal architectures that can model the simultaneous interplay between visual, auditory, and linguistic processing.
- Data Efficiency: By framing brain encoding as an in-context learning problem, Muquan Yu et al. demonstrated in Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex that models can generalize to entirely new subjects and datasets using only a handful of examples, provided the underlying architecture is built to capture cross-modal representations.
Ultimately, grounding AI architectures in biological reality yields superior representations. As Zhisheng Chen et al. showed with Uni-NTFM: A Unified Foundation Model for EEG Signal Representation Learning, incorporating core neuroscience principles—such as decoupled neural coding and sparse functional modularity—allows a 1.9-billion-parameter model to shatter downstream performance records.
To support these complex workflows in practice, the community is building unified software pipelines. For example, Jianwei Shi et al. introduced EPAT: a user-friendly MATLAB toolbox for EEG/ERP data processing and analysis specifically to help researchers bridge the gap between electrical activity and hemodynamic responses (like near-infrared spectroscopy) within a single standardized workflow.
On the hardware side, as outlined by Jinbo Chen et al. in Recent Trends and Future Prospects of Neural Recording Circuits and Systems: A Tutorial Brief, next-generation brain-machine interfaces are moving away from purely electrical acquisition toward integrated platforms capable of simultaneous electrical, chemical, and optical sensing.
As foundation models continue to scale, the integration of these multi-modal recording systems will be essential to transition from fragmented observations to a unified, high-fidelity simulation of the living human brain.
Key Foundational Work
Bayesian brain source imaging based on combined MEG/EEG and fMRI using MCMC (2008) — contributed 1 concept elements to this direction.
Multimodality Data Integration in Epilepsy (2007) — contributed 1 concept elements to this direction.