Abstract
Multi-modal brain networks characterize complex connectivities among brain regions from both structural and functional perspectives. While fusion methods exist, many fail to capture spatio-temporal topological characteristics effectively. We propose MSTGC, an adaptive multi-channel graph convolution network (GCN) framework with graph contrastive learning. MSTGC mines complementary and discriminative features while capturing dynamic characteristics and topological structures. By dividing signals into overlapping time windows and using multi-channel GCNs with InfoMax and InfoMin constraints, the model extracts shared and modality-specific information. Stacked LSTMs then capture temporal transitions. Experiments on an epilepsy dataset demonstrate that MSTGC outperforms state-of-the-art methods in brain disease diagnosis.
The Challenge of Multi-Modal Fusion
Diagnosing brain disorders like epilepsy requires a holistic view of the brain's structure (physical white matter connections via DTI) and function (synchronized activity over time via fMRI). However, effectively fusing these modalities presents significant hurdles:
- Topology Loss: Traditional methods often "flatten" complex brain graphs into simple vectors, destroying the spatial relationships between brain regions.
- Static Bias: Functional connectivity is inherently dynamic, yet many models treat brain activity as a single snapshot, ignoring the rich, time-varying nature of neural chatter.
- Fusion Friction: It is difficult to balance the extraction of unique modality-specific features with the identification of shared complementary information between fMRI and DTI.
The MSTGC Framework
The Multi-Channel Spatio-Temporal Graph Convolution Network (MSTGC) addresses these flaws through a multi-stage pipeline designed to preserve the integrity of brain network data throughout the learning process.
Spatio-Temporal Modeling
The process begins with Dynamic Network Construction, where fMRI signals are sliced into overlapping time windows to create a sequence of graphs. These graphs are then processed by the Adaptive Multi-Channel GCN, which utilizes three parallel channels:
- Specific Channels: Two channels independently extract unique features from fMRI and DTI.
- Common Channel: A parameter-sharing channel identifies overlapping, complementary information between the two modalities.
To capture the evolution of connectivity, the model employs stacked Long Short-Term Memory (LSTM) units. This allows the network to learn how brain communication patterns change over time, which serves as a vital biomarker for identifying epilepsy subtypes.
Graph Contrastive Learning
A key innovation in MSTGC is the use of contrastive constraints to optimize feature extraction:
- Multi-modal fusion InfoMax: Encourages the model to maximize shared information between modalities to find common ground.
- Inter-channel InfoMin: Forces the specific channels to remain distinct, ensuring the model captures unique diagnostic strengths from each modality without redundancy.
Key Results and Insights
The model was evaluated on a dataset of 306 subjects, including patients with Frontal Lobe Epilepsy (FLE), Temporal Lobe Epilepsy (TLE), and healthy controls (NC).
- Superior Diagnostic Performance: MSTGC achieved 81.58% accuracy in distinguishing healthy controls from epilepsy patients, significantly outperforming competitors like SAGPool (78.13%).
- Subtype Differentiation: The model excelled at the difficult task of separating TLE from FLE, reaching 75.59% accuracy, whereas most existing methods struggle to exceed 70%.
- Ablation Success: Removing either the contrastive learning constraints or the LSTM temporal modeling led to a marked decrease in performance, confirming that both spatial topology and temporal dynamics are essential for accurate diagnosis.
Conclusion and Future Directions
MSTGC demonstrates that maintaining graph integrity and modeling temporal dynamics are vital for multi-modal brain analysis. While the model successfully identifies discriminative regions like the Parahippocampal gyrus, future work will focus on improving the stability of feature selection and testing the framework on larger, open-source datasets like the UK Biobank. Additionally, automating the selection of optimal time-window sizes remains a priority for enhancing model robustness.
Summary updated