Learning brain dynamics for decoding and predicting individual differences

Luiz Pessoa, Chirag Limbachia, Joyneel Misra, Srinivas Govinda Surampudi, Manasij Venkatesh, Joseph F. JáJábioRxiv 2021 · DOI

TL;DR

This paper introduces a novel recurrent neural network (RNN) framework for analyzing dynamic brain activity captured by fMRI. The model effectively decodes complex spatiotemporal patterns in brain signals, enabling accurate classification of movie clips and experimental conditions, and even predicting individual differences in cognitive abilities.

What Problem Does This Paper Address?

Traditional fMRI analysis often treats brain activity in a static manner, averaging signals over time. However, cognitive functions are inherently dynamic, evolving in both space and time. This static approach misses crucial information about how brain signals change and interact to support complex behaviors, especially during naturalistic tasks like watching movies. The paper aims to bridge this gap by developing a method that can effectively capture and leverage these dynamic, spatiotemporal brain patterns for decoding and prediction. Specifically, it addresses how to:

  • Characterize generalizable spatiotemporal patterns from dynamic stimuli across participants.
  • Understand the dimensionality of these brain representations.
  • Determine if these dynamic patterns can predict individual differences in cognitive abilities.

Methods

The core of the proposed framework is a Gated Recurrent Unit (GRU)-based neural network architecture. GRUs are a type of RNN well-suited for processing sequential data, allowing the model to learn from past information to inform current predictions.

Key components of their approach include:

  • GRU Classifier: This model takes time-series fMRI data as input and learns to classify stimuli (e.g., movie clips) or experimental conditions at each time step.
  • Dimensionality Reduction (GRU Encoder): To understand the underlying structure of brain dynamics, they employ a GRU encoder that projects the high-dimensional fMRI data into a lower-dimensional latent space. This allows for the visualization of "trajectories" representing brain states over time.
  • GRU Decoder: This component is used to reconstruct the original brain signals from the learned low-dimensional representations, helping to assess what information is preserved.
  • Saliency Maps: To interpret the model's decisions, saliency maps are generated. These maps highlight which brain regions or voxels contribute most to the classification at specific time points, revealing the spatiotemporal importance of different brain areas.
  • Dynamic Multivariate Pattern Analysis (MVPA): By applying the framework at the voxel level, they enable a dynamic version of MVPA, allowing for the analysis of fine-grained spatial patterns over time.

Datasets

The study utilized two primary datasets:

  • Human Connectome Project (HCP) Movie Data: This dataset consists of fMRI scans from 176 participants watching various movie clips. This data was used for training and testing the clip classification and individual difference prediction models.
  • Moving-Circles Paradigm Dataset: This dataset, collected in their lab, involves fMRI data from 61 participants during a continuous experimental paradigm where two circles move on a screen, with stimuli defined as "approach" and "retreat" states. This was used for binary classification at the voxel level.

Key Insights and Ideas

  • Temporal Information is Crucial: Shuffling the temporal order of fMRI data significantly degraded classification accuracy, highlighting that the precise temporal sequence of brain activity is essential for decoding complex stimuli.
  • Low-Dimensional Trajectories Capture Dynamics: The dimensionality reduction approach revealed that brain dynamics can be represented as low-dimensional trajectories in a state space. These trajectories effectively capture the essential information for classification, even with as few as 3-10 dimensions.
  • Saliency Maps Reveal Dynamic Brain Contributions: Saliency maps demonstrated that the importance of different brain regions for classification changes dynamically over time, mirroring the accuracy curve and highlighting specific brain areas involved in processing different aspects of the stimuli.
  • Predicting Individual Differences: The framework successfully predicted individual differences in fluid intelligence and verbal IQ, achieving performance comparable to or better than established methods like connectome-based predictive modeling. This suggests that dynamic brain patterns encode stable individual traits.

Key Results

  • Movie Clip Classification: The GRU classifier achieved ~90% accuracy in a 15-way classification of movie clips at the region-of-interest level.
  • Experimental Condition Classification: For the moving-circles paradigm, the model achieved ~60% accuracy in binary classification of experimental conditions at the voxel level.
  • Individual Difference Prediction: The model demonstrated strong performance in predicting fluid intelligence and verbal IQ scores, with some movie clips showing particularly good predictive power.
  • Dimensionality Reduction Effectiveness: Even with just 3 dimensions, the dimensionality reduction approach achieved 63.30% accuracy in clip prediction, and performance remained high with up to 10 dimensions.

Limitations and Future Work

  • Generalizability of Individual Difference Prediction: While promising, the authors note that the performance for predicting individual differences varied across movie clips and segments. Larger datasets are needed to fully assess this potential.
  • Voxel-Level Dynamic MVPA: The dynamic MVPA at the voxel level achieved modest accuracy (~60%) in the moving-circles paradigm. Future work could explore more complex paradigms or advanced architectures to improve performance.
  • Model Interpretability: While saliency maps provide insights, further work is needed to fully understand the complex, non-linear transformations within the GRU network.
  • Broader Applications: The authors envision their framework being applied to other types of brain data and for discovering and interpreting brain dynamics in various cognitive and clinical contexts. They also suggest that their modular approach allows for the integration of other RNN architectures or novel developments.

Summary updated

Abstract

Abstract Insights from functional Magnetic Resonance Imaging (fMRI), as well as recordings of large numbers of neurons, reveal that many cognitive, emotional, and motor functions depend on the multivariate interactions of brain signals. To decode brain dynamics, we propose an architecture based on recurrent neural networks to uncover distributed spatiotemporal signatures. We demonstrate the potential of the approach using human fMRI data during movie-watching data and a continuous experimental paradigm. The model was able to learn spatiotemporal patterns that supported 15-way movie-clip classification (~90%) at the level of brain regions, and binary classification of experimental conditions (~60%) at the level of voxels. The model was also able to learn individual differences in measures of fluid intelligence and verbal IQ at levels comparable or better than existing techniques. We propose a dimensionality reduction approach that uncovers low-dimensional trajectories and captures essential informational (i.e., classification related) properties of brain dynamics. Finally, saliency maps were employed to characterize brain-region/voxel importance, and uncovered how dynamic but consistent changes in fMRI activation influenced decoding performance. When applied at the level of voxels, our framework implements a dynamic version of multivariate pattern analysis. We believe our approach provides a powerful framework for visualizing, analyzing, and discovering dynamic spatially distributed brain representations during naturalistic conditions. 1 Author summary Brain signals are inherently dynamic and evolve in both space and time as a function of cognitive or emotional task condition or mental state. To characterize brain dynamics, we employed an architecture based on recurrent neural networks, and applied it to functional magnetic resonance imaging data from humans watching movies or during continuous experimental conditions. The model learned spatiotemporal patterns that allowed it to correctly classify which clip a participant was watching based entirely on data from other participants; the model also learned a binary classification of experimental conditions at the level of voxels. We developed a dimensionality reduction approach that uncovered low-dimensional “trajectories” and captured essential information properties of brain dynamics. When applied at the level of voxels, our framework implements a dynamic version of multivariate pattern analysis. We believe our approach provides a powerful framework for visualizing, analyzing, and discovering dynamic spatially distributed brain representations during naturalistic conditions.
BibTeX
@misc{pessoa2021learning,
  author = {Luiz Pessoa and Chirag Limbachia and Joyneel Misra and Srinivas Govinda Surampudi and Manasij Venkatesh and Joseph F. JáJá},
  title = {{Learning brain dynamics for decoding and predicting individual differences}},
  year = {2021},
  doi = {10.1101/2021.03.27.437315},
  howpublished = {bioRxiv},
}
Figure