Brain decoding from functional MRI using long short-term memory recurrent neural networks

Hongming Li, Yong Fan

Abstract

Decoding brain functional states underlying different cognitive processes using multivariate pattern recognition techniques has attracted increasing interests in brain imaging studies. Promising performance has been achieved using brain functional connectivity or brain activation signatures for a variety of brain decoding tasks. However, most of existing studies have built decoding models upon features extracted from imaging data at individual time points or temporal windows with a fixed interval, which might not be optimal across different cognitive processes due to varying temporal durations and dependency of different cognitive processes. In this study, we develop a deep learning based framework for brain decoding by leveraging recent advances in sequence modeling using long short-term memory (LSTM) recurrent neural networks (RNNs). Particularly, functional profiles extracted from task functional imaging data based on their corresponding subject-specific intrinsic functional networks are used as features to build brain decoding models, and LSTM RNNs are adopted to learn decoding mappings between functional profiles and brain states. We evaluate the proposed method using task fMRI data from the HCP dataset, and experimental results have demonstrated that the proposed method could effectively distinguish brain states under different task events and obtain higher accuracy than conventional decoding models.

Introduction

Brain decoding—the ability to determine cognitive states from brain activity—is a cornerstone of modern neuroscience. Traditional approaches often treat functional MRI (fMRI) data as a series of independent snapshots or use fixed-length temporal windows. However, human cognition is inherently sequential; current mental states are deeply influenced by preceding activity. Conventional classifiers like Support Vector Machines (SVM) or Random Forests frequently overlook these temporal dependencies.

To address this, researchers have developed a framework utilizing Long Short-Term Memory (LSTM) recurrent neural networks. LSTMs are uniquely suited for this task as they can capture the temporal flow and varying durations of cognitive processes, significantly outperforming static models in identifying specific tasks.

Methods

The proposed decoding pipeline consists of two primary stages: feature extraction and sequence modeling.

Feature Extraction via Functional Networks

fMRI data is high-dimensional, consisting of thousands of voxels. To reduce dimensionality while preserving biological relevance, the team employed a collaborative sparse brain decomposition technique to identify 90 subject-specific intrinsic functional networks (FNs). Instead of raw voxel data, the model utilizes "functional profiles"—the average activity levels within these 90 networks over time.

Subject-specific intrinsic functional networks used for feature extraction.

Sequence Modeling with LSTMs

These functional profiles are processed by an LSTM RNN architecture. LSTMs maintain an internal state that "remembers" information across sequences. The architecture includes two hidden LSTM layers (256 nodes each), a fully connected layer, and a softmax output for state prediction. To account for the ~6-second hemodynamic delay in fMRI signals, task labels were shifted forward by 8 time points to ensure the model learned the correct associations.

Overview of the LSTM-based brain decoding framework. (a) The end-to-end pipeline from 4D task data to predicted states. (b) The specific LSTM RNN architecture used for classification.

Results

The framework was evaluated using the Human Connectome Project (HCP) dataset, involving 490 healthy adults performing a working memory task (0-back and 2-back tests with various stimuli).

  • Superior Accuracy: The LSTM model achieved a decoding accuracy of 68.7%, surpassing the 62.8% achieved by a Random Forest baseline.
  • State-Specific Performance: The LSTM outperformed traditional models in 5 out of 9 specific brain states, showing particular strength in distinguishing between similar tasks (e.g., different 2-back memory tests).
  • Biological Validity: Sensitivity analysis confirmed that the model prioritized functional networks in the prefrontal and parietal cortices, regions known to be critical for working memory.

Discussion and Conclusion

The primary insight of this work is that temporal context is vital for accurate brain decoding. By capturing the history of neural signals, LSTMs provide a more nuanced understanding of cognitive transitions than static classifiers. Furthermore, using subject-specific networks rather than a generic atlas preserves individual biological variations.

While the model shows promise, limitations remain. Training required cropping data into 40-time-point clips to avoid bias, potentially limiting the learning of very long-term dependencies. Future research will focus on improving model interpretability and testing generalization on naturalistic stimuli, such as movie watching, where temporal context is even more pronounced.

Summary updated

Abstract

Decoding brain functional states underlying different cognitive processes using multivariate pattern recognition techniques has attracted increasing interests in brain imaging studies. Promising performance has been achieved using brain functional connectivity or brain activation signatures for a variety of brain decoding tasks. However, most of existing studies have built decoding models upon features extracted from imaging data at individual time points or temporal windows with a fixed interval, which might not be optimal across different cognitive processes due to varying temporal durations and dependency of different cognitive processes. In this study, we develop a deep learning based framework for brain decoding by leveraging recent advances in sequence modeling using long short-term memory (LSTM) recurrent neural networks (RNNs). Particularly, functional profiles extracted from task functional imaging data based on their corresponding subject-specific intrinsic functional networks are used as features to build brain decoding models, and LSTM RNNs are adopted to learn decoding mappings between functional profiles and brain states. We evaluate the proposed method using task fMRI data from the HCP dataset, and experimental results have demonstrated that the proposed method could effectively distinguish brain states under different task events and obtain higher accuracy than conventional decoding models.
BibTeX
@inproceedings{li2018brain,
  author = {Hongming Li and Yong Fan},
  title = {{Brain decoding from functional MRI using long short-term memory recurrent neural networks}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2018},
  url = {https://openreview.net/forum?id=qWBEBAiwRz},
}
Figure