Semantic representations during language comprehension are affected by context

Fatma Deniz, Christine Tseng, Leila Wehbe, Tom Dupré la Tour, Jack L. GallantbioRxiv 2021 · DOI

TL;DR

This study reveals that the amount of context in linguistic stimuli significantly impacts how the brain represents meaning, affecting both the quality of neuroimaging signals and the specific brain regions involved. The findings underscore the limitations of using decontextualized stimuli in neuroimaging research, suggesting that such studies may not accurately reflect natural language comprehension.

What Problem Does This Paper Address?

Neuroimaging studies investigating how the brain processes word meaning often rely on simplified stimuli like isolated words or sentences, which lack the rich contextual information present in natural language. This raises a critical question: do the findings from these studies, which use "out-of-context" stimuli, generalize to how our brains understand language in real-world scenarios? The authors aim to bridge this gap by directly comparing brain responses to stimuli with varying levels of contextual richness, from single words to full narratives.

Methods

The researchers employed functional Magnetic Resonance Imaging (fMRI) to record brain activity from four subjects while they engaged with linguistic stimuli across four conditions:

  • Narratives: Full stories.
  • Sentences: Sentences sampled from the narratives.
  • Semantic Blocks: Blocks of semantically related words.
  • Single Words: Isolated words.

They then used a voxelwise encoding modeling approach. This involves:

  1. Model Estimation: Building a model for each voxel that predicts brain activity based on features of the stimulus. This model incorporates a 985-dimensional semantic feature space derived from word co-occurrence statistics, along with lower-level linguistic features.
  2. Model Validation: Using a held-out dataset to test how well the estimated models predict brain responses.
  3. Prediction Accuracy: Quantifying the model's performance by calculating the correlation between predicted and actual brain activity.

Additionally, they analyzed the signal-to-noise ratio (SNR) of evoked brain responses, using explainable variance (EV) as a measure of reliability.

Datasets

The study utilized a dataset of ten spoken stories from "The Moth Radio Hour" as the basis for all stimulus conditions. Transcriptions of these stories were used to generate the stimuli. The semantic feature space was constructed using word co-occurrence statistics from a large corpus, including these narratives, books from Project Gutenberg, Wikipedia pages, and Reddit comments.

Key Insights and Ideas

  • Context is King (for Brain Signals): More contextualized stimuli (narratives, sentences) lead to stronger, more reliable brain signals (higher SNR/EV) compared to isolated words or semantic blocks. This suggests that the brain's response is more robust when processing language in a richer context.
  • Widespread Semantic Representation: As context increases, semantic information is represented across a broader network of brain regions, particularly in visual, temporal, parietal, and prefrontal cortices. This highlights that meaning isn't localized to a single spot but distributed across a network that expands with contextual complexity.
  • Individual Variability Matters: While group-level analyses show consistent effects of context, individual subjects exhibit significant variability in how their brains represent semantic information, especially for less natural stimuli. Only natural language stimuli (narratives) consistently evoked widespread semantic representations across all subjects.
  • Generalization Gap: Models trained on stimuli with little context (single words, semantic blocks) do not generalize well to predicting brain activity for natural language stimuli (narratives). This is a crucial finding, implying that research using simplified stimuli might be missing key aspects of natural language processing.
  • Semantic Tuning Shifts: Context influences not only where semantic information is represented but also how it's tuned within specific voxels. This tuning can shift depending on the context, suggesting a dynamic and adaptive semantic processing system.

Key Results

  • SNR Increase: Stimuli with more context (Narratives, Sentences) evoked significantly higher SNR (measured by EV) in bilateral visual, temporal, parietal, and prefrontal cortices compared to stimuli with less context (Single Words, Semantic Blocks). For example, in Subject S1, the number of voxels with significant EV in the Narratives condition was 7622, compared to 256 in the Single Words condition.
  • Widespread Semantic Representation: At the group level, increasing context led to a significant increase in the number of semantically selective voxels. For instance, in the Narratives condition, an average of 4745-7786 voxels were semantically selective per subject, compared to 0-708 for Single Words and Semantic Blocks.
  • Generalization Failure: Models trained on Single Words and Semantic Blocks generalized poorly to predicting brain activity for Narratives. For example, when predicting Narratives activity using Single Words models, only scattered voxels were predicted, with no significant prediction in key temporal, parietal, and prefrontal regions. In contrast, Sentences models showed better generalization.

Limitations and Future Work

  • Subject Number: The study involved only four subjects, which the authors acknowledge limits the generalizability of findings related to individual differences. They suggest that future research with a larger subject pool is needed to further investigate subject-dependent semantic tuning.
  • Semantic Model Limitations: The current semantic model, while powerful, has limitations. It captures some narrative information correlated with word-level semantics, potentially reflecting higher-level linguistic or domain-general representations. Additionally, it uses a static word embedding, failing to differentiate between word senses or contexts.
  • Future Directions: The authors propose exploring other semantic models that explicitly account for contextual information, narrative structure, metaphor, and humor. They also suggest that future studies should prioritize using more naturalistic stimuli to ensure findings generalize to real-world language comprehension.

Summary updated

Abstract

Abstract The meaning of words in natural language depends crucially on context. However, most neuroimaging studies of word meaning use isolated words and isolated sentences with little context. Because the brain may process natural language differently from how it processes simplified stimuli, there is a pressing need to determine whether prior results on word meaning generalize to natural language. fMRI was used to record human brain activity while four subjects (two female) read words in four conditions that vary in context: narratives, isolated sentences, blocks of semantically similar words, and isolated words. We then compared the signal-to-noise ratio (SNR) of evoked brain responses, and we used a voxelwise encoding modeling approach to compare the representation of semantic information across the four conditions. We find four consistent effects of varying context. First, stimuli with more context evoke brain responses with higher SNR across bilateral visual, temporal, parietal, and prefrontal cortices compared to stimuli with little context. Second, increasing context increases the representation of semantic information across bilateral temporal, parietal, and prefrontal cortices at the group level. In individual subjects, only natural language stimuli consistently evoke widespread representation of semantic information. Third, context affects voxel semantic tuning. Finally, models estimated using stimuli with little context do not generalize well to natural language. These results show that context has large effects on the quality of neuroimaging data and on the representation of meaning in the brain. Thus, neuroimaging studies that use stimuli with little context may not generalize well to the natural regime. Significance Statement Context is an important part of understanding the meaning of natural language, but most neuroimaging studies of meaning use isolated words and isolated sentences with little context. Here we examined whether the results of neuroimaging studies that use out-of-context stimuli generalize to natural language. We find that increasing context improves the quality of neuroimaging data and changes where and how semantic information is represented in the brain. These results suggest that findings from studies using out-of-context stimuli may not generalize to natural language used in daily life.
BibTeX
@misc{deniz2021semantic,
  author = {Fatma Deniz and Christine Tseng and Leila Wehbe and Tom Dupré la Tour and Jack L. Gallant},
  title = {{Semantic representations during language comprehension are affected by context}},
  year = {2021},
  doi = {10.1101/2021.12.15.472839},
  howpublished = {bioRxiv},
}

Loading related content...

Figure