BrainLM: A foundation model for brain activity recordings

Josue Ortega, Antonio H. de O. Fonseca, Christopher L. Averill, Syed Asad Rizvi, Matteo Rosati, James M. Cross, Prateek Mittal, Emanuele Zappala, Daniel S. Levine, Rahul M. Dhodapkar, Insu Han, Amin Karbasi, Chadi G. Abdallah, David van Dijk

Abstract

We introduce the Brain Language Model (BrainLM), a foundation model for brain activity dynamics trained on 6,700 hours of fMRI recordings. Utilizing self-supervised masked-prediction training, BrainLM demonstrates proficiency in both fine-tuning and zero-shot inference tasks. Fine-tuning allows for the accurate prediction of clinical variables like age, anxiety, and PTSD as well as forecasting of future brain states. Critically, the model generalizes well to entirely new external cohorts not seen during training. In zero-shot inference mode, BrainLM can identify intrinsic functional networks directly from raw fMRI data without any network-based supervision during training. The model also generates interpretable latent representations that reveal relationships between brain activity patterns and cognitive states. Overall, BrainLM offers a versatile and interpretable framework for elucidating the complex spatiotemporal dynamics of human brain activity. It serves as a powerful “lens” through which massive repositories of fMRI data can be analyzed in new ways, enabling more effective interpretation and utilization at scale.

Introduction

Functional magnetic resonance imaging (fMRI) provides a non-invasive window into the living brain by measuring blood-oxygen-level dependent (BOLD) signals. However, this data is notoriously difficult to work with: it is high-dimensional, noisy, and possesses complex spatiotemporal dynamics that vary significantly between individuals.

Traditionally, researchers have built "narrow" machine learning models designed for specific tasks—like identifying a specific disease or decoding a single cognitive state. These models often fail to generalize to new datasets and struggle to leverage the vast repositories of unlabeled fMRI data available today. BrainLM addresses this by creating a flexible, generative framework that learns the underlying "grammar" of brain activity from massive amounts of data without needing explicit labels.

Overview of the BrainLM framework, illustrating the pipeline from input data through the foundation model to zero-shot inference and fine-tuning tasks.

Methods and Architecture

The researchers adapted the Masked Autoencoder (MAE) architecture—a staple in computer vision and natural language processing—for fMRI time series. The core of BrainLM is a Transformer-based model trained via self-supervision.

  1. Preprocessing: The brain is parcellated into 424 regions (using the AAL-424 atlas), converting 4D fMRI data into 424-dimensional time series.
  2. Self-Supervised Training: During training, the model is "blinded" to a large portion of the data (up to 75% masking ratio). It must then learn to reconstruct the missing segments by attending to the visible parts of the recording. This forces the model to learn the intricate dependencies of how different brain regions interact over time.
  3. In Silico Perturbation: A novel "prompting" technique allows the model to act as a simulator. By nudging the model's latent representations, researchers can observe how the predicted brain state changes, effectively performing "virtual experiments."
The BrainLM architecture details the masked autoencoding process, where input data is masked and then reconstructed using a Transformer-based encoder-decoder structure.

Datasets and Generalization

The model was trained and evaluated on two of the largest fMRI repositories in existence:

  • UK Biobank (UKB): Approximately 42,000 individuals (77,000 recordings, 6,700 hours). This served as the primary training and internal testing set.
  • Human Connectome Project (HCP): 1,002 high-quality recordings used as an independent external validation set to prove the model's ability to generalize across different scanners and populations.

Key Insights

  • Scaling Matters: Training on 1-2 orders of magnitude more data than previous attempts resulted in more robust and generalizable representations.
  • Emergent Functional Organization: The model’s internal self-attention weights naturally aligned with known functional networks (e.g., visual or default mode networks) without any supervision. The model "discovered" the functional organization of the brain on its own.
  • Zero-Shot Versatility: BrainLM can perform "zero-shot" inference to identify functional networks or simulate perturbations without any additional training.
  • A Foundation for Biomarkers: Fine-tuning the pre-trained model proved significantly more effective than training from scratch on raw data for clinical tasks.

Results

  • Generalization: BrainLM achieved an average $R^2$ score of 0.402 on held-out UKB data and 0.316 on the entirely external HCP dataset.
  • Clinical Prediction: Fine-tuning reduced the Mean Squared Error (MSE) for age prediction from 2.0 (raw data) to 0.485, significantly outperforming traditional baselines like SVMs.
  • Forecasting: BrainLM outperformed specialized architectures like LSTMs and Neural ODEs in predicting future brain states (20 timesteps ahead).
  • Functional Identification: The model classified brain regions into functional networks with 58.8% accuracy using internal attention weights, compared to 39.2% using raw data.

Conclusion and Future Work

BrainLM represents a significant shift toward "Big Data" in neuroscience, suggesting that Transformer-based revolutions can help decode the complex spatiotemporal dynamics of the human brain. While successful, future work aims to increase spatial resolution to the voxel level, integrate multi-modal data (EEG, MEG, genomics), and expand testing to a wider array of neurodegenerative and psychiatric conditions.

Summary updated

Abstract

A bstract We introduce the Brain Language Model (BrainLM), a foundation model for brain activity dynamics trained on 6,700 hours of fMRI recordings. Utilizing self-supervised masked-prediction training, BrainLM demonstrates proficiency in both fine-tuning and zero-shot inference tasks. Fine-tuning allows for the accurate prediction of clinical variables like age, anxiety, and PTSD as well as forecasting of future brain states. Critically, the model generalizes well to entirely new external cohorts not seen during training. In zero-shot inference mode, BrainLM can identify intrinsic functional networks directly from raw fMRI data without any network-based supervision during training. The model also generates interpretable latent representations that reveal relationships between brain activity patterns and cognitive states. Overall, BrainLM offers a versatile and interpretable framework for elucidating the complex spatiotemporal dynamics of human brain activity. It serves as a powerful “lens” through which massive repositories of fMRI data can be analyzed in new ways, enabling more effective interpretation and utilization at scale. The work demonstrates the potential of foundation models to advance computational neuroscience research.
BibTeX
@inproceedings{ortega2024brainlm,
  author = {Josue Ortega and Antonio H. de O. Fonseca and Christopher L. Averill and Syed Asad Rizvi and Matteo Rosati and James M. Cross and Prateek Mittal and Emanuele Zappala and Daniel S. Levine and Rahul M. Dhodapkar and Insu Han and Amin Karbasi and Chadi G. Abdallah and David van Dijk},
  title = {{BrainLM: A foundation model for brain activity recordings}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=RwI7ZEfR27},
}
Figure