Modeling the Dynamics of Human Brain Activity with Recurrent Neural Networks

Umut Güçlü, Marcel A. J. van GervenFrontiers Comput. Neurosci 2017 · DOI

TL;DR

This paper introduces recurrent neural networks (RNNs) as a powerful new tool for modeling brain activity in response to naturalistic stimuli. By leveraging RNNs' ability to capture temporal dependencies, the authors demonstrate a significant improvement over traditional methods, offering a more nuanced understanding of how the brain processes dynamic sensory information.

What Problem Does This Paper Address?

Understanding how the human brain represents and processes sensory information is a fundamental goal in neuroscience. "Encoding models" are a key tool for this, aiming to predict brain activity based on sensory input. These models typically consist of two parts: a "feature model" that extracts relevant information from stimuli, and a "response model" that translates these features into predicted brain activity. While much effort has gone into improving feature models (e.g., using deep learning for image recognition), response models have lagged behind. Standard response models often assume a simple linear relationship between features and brain signals, failing to capture the complex temporal dynamics inherent in neural responses, which are influenced by factors like neurovascular coupling and cognitive processing. This leaves a significant portion of brain activity unexplained, preventing us from reaching the "noise ceiling" of prediction accuracy.

Methods

The core innovation of this paper is the application of Recurrent Neural Networks (RNNs) as response models. Unlike traditional linear models, RNNs possess internal "memory" mechanisms that allow them to process sequences of information and capture long-term temporal dependencies. The authors specifically explore two types of RNNs: Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), varying the number of units in their hidden layers (10, 50, and 100). These RNNs are trained end-to-end to predict brain activity from extracted features.

To evaluate their approach, the authors compare these RNN-based response models against a family of ridge regression models. These ridge regression models represent the state-of-the-art in traditional response modeling and include variations that account for hemodynamic delays using canonical hemodynamic response functions (R-C, R-CTD) or finite impulse response models (R-F).

Two types of feature models are used to provide input to the response models:

  • High-level semantic model: Pre-trained Word2Vec (W2V) model, which generates 300-dimensional feature vectors representing semantic content of stimuli.
  • Low-level visual feature model: GIST model, which generates 512-dimensional feature vectors capturing spatial envelope information of visual scenes.

The authors also introduce a method for estimating voxel-specific Hemodynamic Response Functions (HRFs) by stimulating the RNN model with an impulse.

Datasets

The study utilizes the vim-2 dataset, which was originally published by Nishimoto et al. (2011). This dataset consists of:

  • Stimulus: Naturalistic movie clips.
  • Response: Blood-oxygen-level dependent (BOLD) signals recorded via fMRI from the occipital cortices of three subjects.
  • The dataset includes training, validation, and test sets, with the test set containing repeated stimulus presentations for averaging.

Key Insights and Ideas

  • RNNs excel at capturing temporal dynamics: The most significant insight is that RNNs, with their internal memory, are far more adept at modeling the complex temporal dependencies in brain activity than traditional linear models. This allows them to better predict responses to dynamic, naturalistic stimuli.
  • Beyond linear assumptions: The paper challenges the long-standing assumption of linearity in response models, demonstrating that the brain's response to stimuli is inherently non-linear and time-dependent.
  • Feature model choice matters for brain regions: The study shows a dissociation in how well low-level visual features (GIST) and high-level semantic features (W2V) explain brain activity in different visual areas. Low-level features are better predictors in upstream visual areas (like V1), while high-level semantic features are more predictive in downstream areas (like TOS and OFA).
  • RNNs can estimate HRFs: A novel contribution is the demonstration that RNNs can be used to estimate voxel-specific HRFs from continuous fMRI data, a task that has been challenging with traditional methods.
  • Internal representations are key: Analysis of the RNN's internal states reveals that they become increasingly correlated with the predicted brain responses as information flows through the network, suggesting that these internal representations are crucial for capturing the complex relationship between stimuli and brain activity.

Key Results

  • Significant performance improvement: RNN-based response models (specifically G-100 and L-100) consistently and significantly outperformed ridge regression models across various brain regions and feature types. For example, with the W2V model, RNNs achieved a median performance of r=0.16 compared to r=0.12 for the best ridge regression model.
  • Downstream vs. Upstream specialization: When using the W2V model, RNN performance was highest in downstream visual areas (e.g., TOS, OFA), while with the GIST model, performance was highest in upstream areas (e.g., V1, V2).
  • Feature model preference: Approximately 39% of voxels showed a preference for either semantic or perceptual representations, with a notable split in which representation was preferred depending on the brain region.
  • HRF estimation: The estimated HRFs from the RNN model showed a similar global shape to canonical HRFs but revealed considerable variability in response and undershoot delays across voxels.

Limitations and Future Work

  • Data requirements: The authors acknowledge that RNNs, like other deep learning models, require a large amount of training data to avoid overfitting due to their numerous parameters. This can be a limitation when dealing with smaller datasets.
  • Interpretability: While RNNs offer superior predictive power, their complex, non-linear nature can make their internal workings harder to interpret compared to simpler linear models. The paper suggests that future work could focus on developing methods to better understand these internal representations.
  • Feature dimensionality: The paper notes that RNNs might struggle with very high-dimensional stimulus features, suggesting that dimensionality reduction techniques may be necessary in such cases.
  • Model comparison: The authors suggest that further research could explore other RNN architectures and compare them to other advanced feature models. They also propose that RNNs could be used in decoding frameworks, similar to MVPA, to potentially improve performance by accounting for temporal dependencies.

Summary updated

BibTeX
@article{guclu2017modeling,
  author = {Umut Güçlü and Marcel A. J. van Gerven},
  title = {{Modeling the Dynamics of Human Brain Activity with Recurrent Neural Networks}},
  journal = {Frontiers Comput. Neurosci.},
  year = {2017},
  doi = {10.3389/fncom.2017.00007},
}
Figure