Joint Modeling of fMRI and EEG Imaging Using Ordinary Differential Equation-Based Hypergraph Neural Networks

YanZhang, Yang Gao, Min LiNeurIPS 2025 · OpenReview

TL;DR

FE-NET is a novel deep learning framework that jointly models fMRI and EEG data using Neural Ordinary Differential Equations (ODEs) and hypergraphs. By treating brain activity as a continuous dynamical system, it successfully integrates asynchronous data—recorded at different times and sampling rates—to outperform state-of-the-art methods in predicting cognitive traits and neurological states.

The Challenge of Fusing fMRI and EEG

Fusing functional Magnetic Resonance Imaging (fMRI) and Electroencephalography (EEG) is considered a "holy grail" in neuroimaging. fMRI offers high spatial resolution but is slow (measuring blood flow over seconds), while EEG offers millisecond-level temporal resolution but poor spatial localization.

However, two major hurdles have historically stalled progress in joint modeling:

  1. The Data Scarcity & Cost Gap: Simultaneously recorded fMRI-EEG datasets are small, highly constrained, and expensive to produce. Asynchronous datasets (where the same subject undergoes both scans, but at different times) are much larger and easier to acquire, but notoriously difficult to align.
  2. The Modality Mismatch: fMRI and EEG measure fundamentally different physical processes (hemodynamics vs. electrical oscillations). Furthermore, the relationships between brain regions are rarely one-to-one; they often involve complex, many-to-many interactions that standard graph-based models fail to capture.

Proposed Methodology: FE-NET

The authors introduce FE-NET, a framework consisting of two primary modules designed to bridge the gap between these disparate signals:

Architecture of the FE-NET framework, detailing the FEH hypergraph generation module and the FED dynamic embedding layer.
  • GAN-based Hypergraph Generation (FEH): Instead of simple pairwise graphs, the model uses hypergraphs, where a single "hyperedge" can connect multiple brain regions (ROIs). This module uses a Generative Adversarial Network (GAN) architecture featuring a novel Optimal fMRI-EEG Isomorph algorithm (OFEI). This ensures that the structural representations of the brain are topologically consistent across both fMRI and EEG modalities.
  • Dynamic Hypergraph Embedding (FED): To handle the timing mismatch, the authors employ Neural ODEs. Rather than processing data in discrete steps (like an RNN), Neural ODEs model the continuous evolution of brain states. This allows the model to naturally accommodate the different sampling rates of EEG and fMRI and capture latent temporal dependencies that discrete models miss.

The framework essentially "learns" the differential equations that govern how brain activity transitions from one state to another, allowing it to fuse slow-moving fMRI patterns with rapid EEG spikes.

Datasets

The study utilizes two key datasets to validate the approach:

  • LEMON Dataset: A large, open-access asynchronous dataset containing 227 healthy participants (young and elderly). This served as the primary benchmark for predicting cognitive traits like attention, intelligence, and flexibility.
  • CN-EPFL Dataset: A smaller, synchronized dataset (20 individuals) used specifically to evaluate the model's ability to generalize to simultaneously recorded data.

Key Insights

  • Closing the "Modality Gap": One of the most striking findings is the visualization of the embedding space. In traditional models, fMRI and EEG data points cluster into two distinct "islands." FE-NET significantly reduces this distance, forcing the representations into a shared space that reflects underlying neural truth rather than modality-specific noise.
  • Hypergraphs > Graphs: By using hyperedges, the model identified discriminative functional connectivity patterns within the Default Mode Network (DMN) and Ventral Attention Network (VAN). These higher-order connections were more predictive of verbal and logical intelligence than simple pairwise correlations.
  • Continuous vs. Discrete: The authors found that Neural ODEs outperformed standard LSTMs and GRUs. This suggests that brain dynamics are better represented as a smooth, continuous flow rather than a sequence of snapshots, especially when trying to align signals with vastly different "clocks."

Experimental Results

FE-NET demonstrated superior performance across four major cognitive classification tasks (predicting High vs. Low scoring groups):

Performance comparison of FE-NET against state-of-the-art methods across various cognitive classification tasks.
  • Sustained Attention: Achieved 85.29% accuracy, significantly outperforming the next best multimodal model, MMP-GCN (~79%).
  • Cognitive Flexibility: Reached 79.18% accuracy.
  • Intelligence: Scored 82.19% (Verbal) and 77.38% (Logical) accuracy.
  • Ablation Success: Removing either the FEH (hypergraph) or FED (ODE) modules led to a sharp drop in performance (roughly 5-8%), proving that both the structural alignment and the continuous temporal modeling are essential for the framework's success.

Limitations and Future Directions

While the results are highly promising, the paper notes a few areas for improvement:

  • Computational Complexity: Training both GANs and Neural ODEs is computationally intensive. Scaling this architecture to even higher-resolution brain atlases may present hardware challenges.
  • Interpretability: While the model identifies important hyperedges, the "black box" nature of Neural ODEs makes it difficult to pinpoint the exact causal mechanisms of the learned differential equations.
  • Dataset Diversity: The study focused on healthy participants and specific cognitive tests. Future work is needed to determine if FE-NET maintains its efficacy in clinical settings for diagnosing neurological disorders like Alzheimer’s or epilepsy.

Summary updated

Abstract

Fusing multimodal brain imaging has been a hot topic since different modalities of brain imaging can provide complementary information. However, due to the size of simultaneous recorded fMRI-EEG dataset being limited and the substantial discrepancy between hemodynamic responses of fMRI and neural oscillations of EEG, the joint modeling of fMRI and EEG images is a rarely explored area and has not yielded satisfactory results. Existing studies have also indicated that the relationships between region of interest (ROI) are not one-to-one when synchronizing fMRI and EEG. Current graph-based multimodal modeling methods overlook those information. Based on this, we propose a hypergraph based fMRI-EEG modeling framework for asynchronous fMRI-EEG data named FE-NET. To the best of our knowledge, this is the first attempt to jointly model asynchronous EEG and fMRI data as Neural ODEs based hypergraph. Extensive experiments have demonstrated that the proposed FE-NET outperforms many state-of-the-art brain imaging modeling methods. Meanwhile, compared to simultaneously recorded fMRI-EEG data, asynchronously acquired fMRI-EEG data is less costly, which demonstrates the practical applicability of our method.
BibTeX
@inproceedings{yanzhang2025joint,
  author = {YanZhang and Yang Gao and Min Li},
  title = {{Joint Modeling of fMRI and EEG Imaging Using Ordinary Differential Equation-Based Hypergraph Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=qJLPlZSdkb},
}
Figure