Incorporating Context into Language Encoding Models for fMRI

Shailee Jain, Alexander HuthNeurIPS 2018 · OpenReview

Introduction

Traditional language encoding models in neuroscience typically rely on static word embeddings like word2vec. While powerful, these models treat every word as an independent unit, effectively viewing language as a "bag-of-words." This ignores the fundamental way humans process speech: we use previous words—or context—to resolve homonyms, parse complex syntax, and track narrative threads.

This paper demonstrates that fMRI encoding models, which predict brain activity from language stimuli, perform significantly better when they incorporate linguistic context using Long Short-Term Memory (LSTM) networks. By moving beyond static word embeddings to contextual representations, the researchers show that the human brain’s "temporal receptive fields" vary across the cortex, with higher-level areas requiring longer context windows to be accurately modeled.

Methodology

The challenge in modeling context is that it is difficult to hand-design as a feature space. To address this, the authors developed a contextual language encoding model using a self-supervised deep learning pipeline to automatically extract rich, contextual features.

Contextual language encoding model pipeline

The pipeline consists of three main steps:

  1. Feature Extraction: A 3-layer LSTM language model was pre-trained on a large corpus of Reddit comments (over 20 million words). For any given word in a narrative story, the researchers extracted a 985-dimensional vector from the LSTM layers using a specific "context length" (the number of preceding words the LSTM was allowed to see, ranging from 0 to 19).
  2. Signal Alignment: Because fMRI (BOLD) responses are much slower than speech, a Lanczos filter was used to downsample the high-frequency word representations to the low-frequency fMRI acquisition rate. A Finite Impulse Response (FIR) model also accounted for the hemodynamic delay (the 4–8 second lag between neural activity and blood flow changes).
  3. Voxelwise Modeling: Ridge regression mapped these contextual features to the activity of individual brain voxels. By comparing the prediction accuracy of these models against a baseline "embedding-only" model, the team quantified the value of context.

The fMRI data used for evaluation consisted of brain responses from 6 subjects listening to 11 naturally spoken narrative stories from The Moth Radio Hour (approximately 2 hours of audio).

Key Findings

The results strongly indicate that the brain's representation of a word is deeply intertwined with its history in the conversation. Contextual models outperformed static embedding models in nearly every area of the brain involved in language.

Encoding model performance across different context lengths and distortion conditions
  • Context Length and Temporal Receptive Fields: Contextual models showed a monotonic increase in prediction accuracy as context length increased, eventually plateauing at around 10–15 words. The study mapped "context sensitivity" across the cortex, revealing a hierarchy: low-level areas like the Auditory Cortex prefer short context, while higher-level semantic areas (like the temporal-parietal junction) prefer long context.
  • The "Middle Layer" Mystery: In supervised vision models, the highest layers usually map to the highest brain areas. However, in this self-supervised language model, Layer 2 (the middle layer) provided the best overall predictions. The authors suggest that in self-supervised models, the "highest-level" representations might emerge in the middle, while the final layer remains more tied to the specific task of next-word prediction.
  • Validation via Distortion: To prove the model was using actual linguistic context rather than just benefiting from a "better" embedding space, the researchers tested models with "random" context (nonsensical word strings) and "swapped" context (plausible but incorrect context). In both cases, performance dropped sharply, confirming the importance of the specific narrative sequence.
  • Language Modeling Correlation: There was a strong correlation between the LSTM’s ability to predict the next word (inverse perplexity) and its ability to predict brain activity. As AI language models get better at "understanding" text, they become better models of the human brain.

Limitations and Future Directions

While the performance plateaued at 15 words, it remains unclear if this is a limitation of the LSTM architecture—which can struggle with very long-range dependencies—or a reflection of the human brain's actual integration window for these specific stimuli. Furthermore, given the current dominance of Transformer architectures, a natural next step is to investigate whether attention mechanisms provide an even better fit for cortical representations. Finally, fine-tuning the language model on the stimulus stories yielded no benefit, likely due to the limited size of the fMRI stimulus set, suggesting larger datasets might be needed for specialized brain-language alignments.

Summary updated

Abstract

Language encoding models help explain language processing in the human brain by learning functions that predict brain responses from the language stimuli that elicited them. Current word embedding-based approaches treat each stimulus word independently and thus ignore the influence of context on language understanding. In this work we instead build encoding models using rich contextual representations derived from an LSTM language model. Our models show a significant improvement in encoding performance relative to state-of-the-art embeddings in nearly every brain area. By varying the amount of context used in the models and providing the models with distorted context, we show that this improvement is due to a combination of better word embeddings learned by the LSTM language model and contextual information. We are also able to use our models to map context sensitivity across the cortex. These results suggest that LSTM language models learn high-level representations that are related to representations in the human brain.
BibTeX
@inproceedings{jain2018incorporating,
  author = {Shailee Jain and Alexander Huth},
  title = {{Incorporating Context into Language Encoding Models for fMRI}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=Egvgj567-Po},
}
Figure