Bridging LSTM Architecture and the Neural Dynamics during Reading

Peng Qian, Xipeng Qiu, Xuanjing Huang

Abstract

This study explores the cognitive plausibility of Long Short-Term Memory (LSTM) networks by establishing a direct mapping between their internal computational states and real-time human brain activity. By aligning an LSTM’s memory cells with fMRI data from subjects engaged in continuous reading of Harry Potter, the researchers demonstrate that artificial "memory vectors" can predict sequential neural dynamics with up to 86% accuracy. These results provide significant evidence that the architectural components of LSTMs—specifically their ability to gate and store long-term context—correlate with the biological mechanisms of language processing in the human brain.

Introduction

While LSTMs have become a cornerstone of sequence modeling in artificial intelligence, their design was originally informed by only a loose biological inspiration. Despite their success in tasks like machine translation and speech recognition, the extent to which their internal mechanics mirror human cognitive processes remains an open question. Most prior research in "brain decoding" has focused on isolated words or static representations. This paper addresses the challenge of sequential reading, investigating whether the way an LSTM manages long-term context aligns with the neural activity of a human reading a continuous narrative.

Methodology

The researchers developed a "brain-LSTM mapping" framework to bridge artificial and biological neural networks. First, a standard LSTM language model was trained on the text of Harry Potter and the Philosopher’s Stone (excluding the test chapter). Second, the study utilized fMRI data from eight subjects reading Chapter 9 of the same book. Given the low temporal resolution of fMRI (one image every 2 seconds), the LSTM states were aligned with brain activity corresponding to every four words read.

Figure 1: Overview of the experimental pipeline, aligning linguistic stimuli with LSTM states and fMRI BOLD signals.

A linear mapping function was employed to determine if the LSTM’s internal vectors ($a_t$) could predict observed BOLD (blood-oxygen-level-dependent) signals ($y_t$). The study also included ablation experiments, systematically disabling specific gates (input, output, or forget) to evaluate their individual contributions to the model's biological alignment.

Key Results and Insights

The study yielded several critical findings regarding the relationship between AI architectures and neural dynamics:

  • The Primacy of the Memory Cell: The LSTM’s memory vector ($c_t$) proved to be a significantly more accurate predictor of brain activity than the hidden state vector ($h_t$). This suggests that the brain’s working memory during reading functions more like the LSTM's internal storage cell than its immediate output state.
  • Contextual Integration: Prediction accuracy was highly dependent on the length of the linguistic context provided to the model. Accuracy dropped sharply with small word windows (e.g., 4 or 8 words) and peaked when the model had access to larger contexts, mirroring the human ability to integrate information across sentences and paragraphs.
Figure 4: Similarity results for memory cell $c_{(t)}$ (a) and hidden state $h_{(t)}$ (b) across various training and testing window sizes.
  • The Role of the Forget Gate: Surprisingly, removing the forget gate improved the model’s correlation with the brain in this specific narrative task. The authors suggest that because reading a story is a continuous, additive process, the brain may not "clear" its context as frequently as a standard forget gate mechanism might imply.
  • Anatomical Correlation: The LSTM’s representations showed the strongest correlation with brain regions associated with language processing and semantic memory, including the prefrontal cortex, temporal cortex, and the Visual Word Form Area (VWFA).

Discussion and Limitations

The memory vector achieved a normalized similarity score of 0.862, vastly outperforming standard Recurrent Neural Networks (RNNs), which scored only 0.508. This highlights the necessity of gating mechanisms for capturing human-like neural dynamics.

However, the study notes several limitations. The temporal resolution of fMRI is inherently slow; future research using MEG or EEG could provide better insights into the millisecond-scale timing of reading. Additionally, the use of a linear mapping function may oversimplify the relationship between AI and the brain. Finally, the finding regarding the forget gate may be task-specific to continuous narrative reading and might not generalize to tasks requiring frequent context switching.

Conclusion

This research provides a rare bridge between deep learning architectures and cognitive neuroscience. By demonstrating that LSTM memory cells can accurately reconstruct human brain activity during reading, the study validates the biological plausibility of gated recurrent architectures and opens new avenues for understanding how both artificial and biological systems process complex, sequential information.

Summary updated

BibTeX
@inproceedings{qian2016bridging,
  author = {Peng Qian and Xipeng Qiu and Xuanjing Huang},
  title = {{Bridging LSTM Architecture and the Neural Dynamics during Reading}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
  url = {https://openreview.net/forum?id=rk-7uVMOZH},
}
Figure