TL;DR
This paper introduces fMRI-LM, a novel foundation model that bridges functional MRI (fMRI) data with large language models (LLMs). By creating a unique "fMRI-text" corpus and a three-stage framework, fMRI-LM can interpret brain activity in a language-aligned manner, paving the way for more universal understanding of fMRI data.
What Problem Does This Paper Address?
While LLMs have excelled at understanding various modalities like images and audio, applying them to brain imaging, specifically fMRI, has been a significant challenge. Existing fMRI models often require task-specific tuning and labeled data, limiting their scalability and generalizability. Furthermore, there's a lack of natural fMRI-text pairs, making it difficult to ground brain activity in semantic meaning. This paper aims to bridge this gap by developing a universal foundation model for fMRI that can understand brain activity through the lens of language, enabling cross-modal reasoning and richer interpretations.
Methods
fMRI-LM employs a three-stage framework:
- fMRI Tokenizer: This stage uses a Transformer-based encoder and a vector quantizer to convert fMRI signals into discrete tokens. These tokens are then embedded into a space that aligns with the LLM's text embedding space. This is achieved through a combination of reconstruction, domain-adversarial, and contrastive alignment losses.
- LLM Fine-tuning: A pretrained LLM is fine-tuned to jointly model the fMRI tokens and text. This allows the LLM to treat brain activity as a sequence that can be temporally predicted and linguistically described. This stage utilizes a novel, large-scale descriptive corpus that translates fMRI imaging features into structured textual descriptors.
- Multi-Task Multi-Paradigm Instruction Tuning: The model is further trained with diverse instruction-tuning objectives to endow it with high-level semantic understanding. This stage supports various downstream applications by using single-question single-answer, multi-question multi-answer, and open-ended description paradigms.
Datasets
The paper utilizes several large-scale public datasets for training and evaluation:
- UK Biobank (UKB): Used for pretraining and downstream evaluation.
- Adolescent Brain Cognitive Development (ABCD): Used for pretraining and downstream evaluation.
- HCP (Human Connectome Project): Used for downstream evaluation.
- HCP-Aging: Used for downstream evaluation.
- ADNI (Alzheimer's Disease Neuroimaging Initiative): Used for downstream evaluation.
- ADHD200: Used for downstream evaluation.
- ABIDE2 (Autism Brain Imaging Data Exchange II): Used for downstream evaluation.
A key contribution is the creation of a large descriptive corpus that translates fMRI imaging-based features (functional connectivity, functional gradients, graph-theoretical metrics, and ICA components) into structured, caption-like text. This corpus serves as a crucial bridge for aligning fMRI with language.
Key Insights and Ideas
- Bridging the fMRI-Language Gap: The core idea is that LLMs can provide a powerful semantic prior for understanding fMRI signals, enabling richer interpretations than task-specific models.
- Synthetic fMRI-Text Corpus: The creation of a descriptive corpus is crucial for aligning fMRI with language, especially in the absence of natural fMRI-text pairs. This corpus acts as a "caption" for brain activity.
- Universal Foundation Model: The goal is to move beyond task-specific models towards a single, adaptable model that can handle diverse fMRI understanding tasks.
- Parameter-Efficient Tuning: The successful application of LoRA (Low-Rank Adaptation) demonstrates that effective fMRI understanding can be achieved with significantly fewer tunable parameters, making the approach more scalable and efficient.
Key Results
fMRI-LM demonstrates strong performance across various benchmarks:
- State-of-the-Art Performance: fMRI-LM achieves the best or second-best performance on most classification and regression tasks compared to supervised and foundation baselines. For example, it achieves 94.89% accuracy on UKB sex prediction.
- Strong Generalization: The model shows robust zero-shot and few-shot generalization capabilities, performing comparably to models trained with full downstream data even with just a few examples.
- Versatility: The multi-task, multi-paradigm instruction tuning enables fMRI-LM to handle diverse tasks and question formats, including open-ended questions, with impressive accuracy (e.g., comparable to structured paradigms).
- Efficiency: Parameter-efficient tuning via LoRA maintains or improves performance while significantly reducing computational demands.
Limitations and Future Work
- Reasoning Process: A key limitation is the current inability to fully capture or evaluate the model's reasoning process over fMRI data. Ideally, the model should be able to explicitly identify the brain substrates supporting a given diagnosis or prediction, offering more transparent explanations.
- Instruction-Tuning Datasets: Future work will require more rigorous construction of instruction-tuning datasets, including carefully curated and validated annotations, to minimize factual errors and support reliable, interpretable reasoning.
- Biomarker Availability: The authors note that imaging biomarkers are not always fully revealed in the medical literature, which can make it challenging to examine the correctness of the model's reasoning.
Summary updated