Capturing brain dynamics: latent spatiotemporal patterns predict stimuli and individual differences

Manasij Venkatesh, Joseph F. JáJá, Luiz PessoabioRxiv 2020 · DOI

TL;DR

This paper introduces a novel LSTM-based framework to analyze dynamic fMRI data, revealing that brain activity during naturalistic stimuli (movie watching) forms complex spatiotemporal patterns. These patterns are not only generalizable across individuals but also predictive of their behavior and personality traits, outperforming traditional static analysis methods.

What Problem Does This Paper Address?

Traditional analysis of fMRI data often treats it in a quasi-static manner, averaging activity over time. However, the brain is inherently dynamic, with cognitive, emotional, and motor functions relying on the intricate interplay of neuronal populations across space and time. This approach aims to bridge this gap by developing methods to effectively capture and characterize these dynamic spatiotemporal properties of brain events, moving beyond static representations to understand how information is processed and represented over time.

Methods

The core of the proposed method is a Long Short-Term Memory (LSTM) network architecture. LSTMs are well-suited for sequential data like fMRI time series, as their gating mechanisms allow them to learn long-term dependencies. The authors employ an LSTM classifier for predicting movie clip labels from fMRI data.

To address interpretability and dimensionality, they introduce an LSTM encoder that projects high-dimensional LSTM hidden states into a lower-dimensional space, creating "trajectories." This is achieved using a dimensionality reduction (DR-FC) fully-connected layer. They also develop an LSTM decoder to reconstruct the original time series from these low-dimensional representations.

For understanding brain region importance, saliency maps are used to identify which brain regions contribute most to the classification at specific time points. Baseline models, including feed-forward networks and Temporal Convolutional Networks (TCNs), are used for comparison.

Datasets

The primary dataset used is from the Human Connectome Project (HCP), specifically their movie-watching fMRI data. This dataset includes recordings from 176 participants who watched various movie excerpts. The data was preprocessed using standard HCP pipelines. No new benchmarks were introduced, but the study leverages the rich, dynamic nature of the HCP movie-watching data to test their methods.

Key Insights and Ideas

  • Brain dynamics are key: Spatiotemporal patterns in fMRI data are not just noise; they contain rich information about the stimuli being processed and are generalizable across individuals.
  • LSTMs excel at capturing temporal dependencies: LSTMs significantly outperform static feed-forward networks and even TCNs in classifying movie clips, highlighting the importance of long-term temporal relationships in brain activity.
  • Low-dimensional representations retain crucial information: Even when drastically reduced to a few dimensions, LSTM-derived latent representations (trajectories) capture essential discriminative information for clip prediction, suggesting a structured, low-dimensional manifold underlying brain dynamics.
  • Individual differences are encoded in dynamics: These spatiotemporal brain patterns are not only predictive of the external stimuli but also of an individual's unique behavioral and personality traits, suggesting that our internal states leave dynamic signatures in brain activity.
  • Brain region importance is dynamic: Saliency maps reveal that the contribution of different brain regions to predicting stimuli or traits changes over time, emphasizing the dynamic nature of neural processing.

Key Results

  • Clip Classification: The LSTM classifier achieved an impressive 87.35% accuracy for 15-way classification of movie clips, demonstrating generalizability across participants.
  • Temporal Information: Temporal Convolutional Networks (TCNs) with kernel widths of 10 achieved 65.04% accuracy, while LSTMs achieved 87.35%, underscoring the advantage of capturing long-term dependencies. Shuffling temporal order drastically reduced accuracy to 64.29%.
  • Low-Dimensionality: Projecting LSTM outputs to just 3 dimensions still yielded 77.30% accuracy, and with 10 dimensions, accuracy was 87.23%, very close to the full dimensionality. This suggests that latent representations with as few as 10 dimensions capture essential discriminative information.
  • Behavior and Personality Prediction: LSTMs significantly outperformed Connectome-Based Predictive Modeling (CPM) in predicting various behavioral and personality traits (fluid intelligence, verbal IQ, NEO-FFI dimensions), achieving higher Spearman's rank correlations. For instance, LSTM accuracy for Verbal IQ was 0.84 compared to CPM's 0.64 using the top/bottom 10% scorers.

Limitations and Future Work

  • Data limitations for individual differences: The study notes that for predicting behavior and personality, they focused on the top/bottom 10% (and later 30%) of scorers due to limited data, which might affect the generalizability of these specific findings.
  • Reconstruction fidelity: While low-dimensional representations capture predictive information, the LSTM decoder could only reconstruct a modest amount of signal variance (<10%) from these representations, suggesting a loss of fine-grained temporal detail during dimensionality reduction.
  • Future Directions: The authors suggest that their LSTM framework is broadly applicable to other types of dynamic brain data (beyond fMRI) for discovering and interpreting brain dynamics. They also highlight the need to understand the interaction between stimuli- and individual-related components in neural responses and to explore the intrinsic dimensionality of these dynamics further.

Summary updated

Abstract

Abstract Insights from functional Magnetic Resonance Imaging (fMRI), and more recently from recordings of large numbers of neurons through calcium imaging, reveal that many cognitive, emotional, and motor functions depend on the multivariate interactions of neuronal populations. To capture and characterize spatiotemporal properties of brain events, we propose an architecture based on long short-term memory (LSTM) networks to uncover distributed spatiotemporal signatures during dynamic experimental conditions 1 . We demonstrate the potential of the approach using naturalistic movie-watching fMRI data. We show that movie clips result in complex but distinct spatiotemporal patterns in brain data that can be classified using LSTMs (≈ 90% for 15-way classification), demonstrating that learned representations generalized to unseen participants. LSTMs were also superior to existing methods in predicting behavior and personality traits of individuals. We propose a dimensionality reduction approach that uncovers low-dimensional trajectories and captures essential informational properties of brain dynamics. Finally, we employed saliency maps to characterize spatiotemporally-varying brain-region importance. The spatiotemporal saliency maps revealed dynamic but consistent changes in fMRI activation data. We believe our approach provides a powerful framework for visualizing, analyzing, and discovering dynamic spatially distributed brain representations during naturalistic conditions.
BibTeX
@misc{venkatesh2020capturing,
  author = {Manasij Venkatesh and Joseph F. JáJá and Luiz Pessoa},
  title = {{Capturing brain dynamics: latent spatiotemporal patterns predict stimuli and individual differences}},
  year = {2020},
  doi = {10.1101/2020.06.11.146969},
  howpublished = {bioRxiv},
}
Figure