Neuroscience faces a fundamental "uncertainty principle" of measurement. If you want to know exactly where something is happening in the brain, you use functional Magnetic Resonance Imaging (fMRI), which provides millimeter-level spatial resolution but captures signals over seconds. If you want to know exactly when something is happening, you use Electroencephalography (EEG) or Magnetoencephalography (MEG), which capture millisecond-level spikes but struggle to pinpoint their anatomical origin.
The research direction emerging across these papers seeks to break this trade-off. By fusing these disparate modalities through deep learning, researchers are moving toward a "best of both worlds" representation: brain mapping that is both spatially precise and temporally fluid.
The Geometry of Brain Alignment
The primary challenge in multimodal fusion is that EEG and fMRI do not "speak the same language." fMRI measures hemodynamic changes (blood flow) across a 3D grid of voxels, while EEG measures electrical potentials across a 2D array of sensors on the scalp.
To reconcile these, researchers like Ce Ju and Reinmar J. Kobler have moved away from simple statistical correlations toward geometric deep learning. In Deep Geodesic Canonical Correlation Analysis for Covariance-Based Neuroimaging Data, they treat brain activity as points on a Symmetric Positive Definite (SPD) manifold. By calculating "geodesic correlations"—essentially finding the shortest path between two points on a curved surface—they can align the underlying geometry of electrical and metabolic signals without forcing them into a flat, Euclidean space that would distort their relationship.
Bridging the Sampling Gap with Dynamical Systems
Even if the signals are aligned geometrically, they are sampled at vastly different rates. EEG might record 1,000 snapshots per second, while fMRI records one every two seconds. Traditional models often downsample the fast signal or average the slow one, losing the very resolution they aim to preserve.
A more sophisticated approach, exemplified by FE-NET from YanZhang and Min Li, treats the brain as a continuous dynamical system. By using Neural Ordinary Differential Equations (ODEs), the model learns the "flow" of brain activity over time. This allows the system to integrate asynchronous data—filling in the gaps between fMRI scans with the high-frequency trajectories provided by EEG.
Generative Super-Resolution and Decoding
If we can learn the relationship between "where" and "when," we can use one modality to "hallucinate" the missing resolution of another. This is the logic behind EEG super-resolution. Shuqiang Wang and Yanyan Shen introduced STAD, a diffusion-based model that takes low-resolution EEG data and "upscales" it to 256 channels.
This generative capability is most visible in "mind-reading" tasks—reconstructing visual images from brain waves. Because EEG signals are notoriously noisy, models like BrainVis and NICE use contrastive learning to align neural signals with high-level semantic features from models like CLIP. Yohann Benchetrit and colleagues demonstrated that by leveraging latent diffusion, it is possible to reconstruct not just the category of an object a person is seeing, but its specific visual appearance, using the millisecond-by-millisecond timing of MEG to track how the "aha!" moment of recognition unfolds in the brain.
From Graphs to Foundation Models
The field is currently shifting toward universal representations. Rather than building a new model for every task, researchers are developing "Brain Foundation Models." BrainGFM, developed by Xinxu Wei and kanhao zhao, treats the brain as a graph where nodes (regions) and edges (connectivity) can be analyzed regardless of which "atlas" or scanning technology was used.
By pre-training on massive multimodal datasets like Alljoined, these models learn the fundamental "grammar" of neural activity. This allows them to identify biomarkers for complex disorders like Schizophrenia or Depression—as seen in the work on structure-revealing fusion by Evrim Acar—by looking for patterns that persist across both the spatial maps of fMRI and the temporal oscillations of EEG.