The Language Network Is Recruited but Not Required for Nonverbal Event Semantics

Anna A. Ivanova, Zachary Mineroff, Vitor Zimmerer, Nancy Kanwisher, Rosemary Varley, Evelina FedorenkoNeurobiology 2021 · DOI

TL;DR

This study investigates whether the brain's language network is essential for understanding complex events, even when presented visually. Using fMRI in healthy adults and studying individuals with severe language impairments (global aphasia), the researchers found that while the language network is active during nonverbal event understanding, it's not necessary. This suggests that our ability to make sense of events is not solely dependent on language.

What Problem Does This Paper Address?

For a long time, there's been a debate about the relationship between language and thought. One prominent idea is that language is crucial for forming complex thoughts, particularly for combining individual concepts into meaningful representations of events – like understanding "who did what to whom." This suggests that the brain's language network, typically located in the left frontal and temporal lobes, might be essential for this kind of "combinatorial semantic processing." However, it's also known that we can understand concepts without language (e.g., recognizing a cat from a picture). This paper aims to resolve this by asking: Is the language network involved in understanding nonverbal events, and more importantly, is it necessary for it?

Methods

The researchers employed a two-pronged approach:

  1. fMRI in Neurotypical Participants (Experiment 1):
    • Participants viewed either sentences or line drawings depicting simple agent-patient interactions.
  2. They performed two tasks: a semantic plausibility judgment (is this event plausible?) and a perceptual control task (is it moving left or right?). The perceptual task was designed to be difficult and not rely on semantic understanding.
  3. The researchers identified individual language functional regions of interest (fROIs) using a standard language localizer task (sentences vs. nonwords).
  4. They then measured the blood-oxygen-level dependent (BOLD) response in these fROIs during the semantic and perceptual tasks for both sentences and pictures.
  5. Neuropsychological Study with Global Aphasia Patients (Experiment 2):
    • Two individuals with global aphasia (severe damage to language areas) and age-matched control participants performed two tasks:
  6. Picture plausibility judgment: Identical to the semantic task in Experiment 1, using only pictures.
  7. Sentence-picture matching: Participants saw a picture and a sentence that either correctly or incorrectly described the agent-patient relationship.
  8. This experiment aimed to see if individuals with severely impaired language networks could still perform nonverbal semantic tasks.

Datasets

The study used custom-created stimuli:

  • Sentences: Describing plausible and implausible agent-patient interactions.
  • Line Drawings: Depicting plausible and implausible agent-patient interactions. These were carefully controlled to prevent reliance on low-level visual cues.
  • Nonwords: Used in the language localizer task.

No pre-existing large-scale datasets were used for training or evaluation in the traditional machine learning sense. The stimuli were designed specifically for the experimental paradigms.

Key Insights and Ideas

  • Language Network is "Recruited" but Not "Required": This is the central finding. The language network does show increased activity when healthy individuals process the meaning of events, whether presented verbally or visually. However, individuals with severe damage to this network can still perform these nonverbal semantic tasks, demonstrating that the network isn't essential.
  • Verbal vs. Nonverbal Processing Differences: While the language network is engaged by both sentences and pictures, the response is stronger for sentences. This hints that language might play a more dominant or efficient role in semantic processing, even if not strictly necessary.
  • Dissociation Between Language and Nonverbal Semantics: The study provides strong evidence for a functional separation between the brain systems supporting language and those supporting nonverbal understanding of events. This challenges theories that posit language as a prerequisite for complex thought.
  • Importance of Event-Level Semantics: The research moves beyond studying single object semantics to understanding how we process complex interactions and relationships between entities in a scene.

Key Results

  • Experiment 1 (fMRI):
    • The language network showed significantly stronger responses during the semantic task (plausible/implausible judgment) compared to the perceptual control task, for both sentences and pictures.
  • The semantic task effect was stronger for sentences than for pictures.
  • This engagement was observed across multiple language-responsive regions (fROIs), including the inferior frontal gyrus (IFG), middle frontal gyrus (MFG), anterior and posterior temporal lobes, and the angular gyrus (AngG).
  • Experiment 2 (Aphasia Patients):
    • Picture Plausibility Judgment: Individuals with global aphasia performed well above chance, comparable to controls, in judging the plausibility of depicted events. Their accuracy was high (around 85-91%), only slightly below controls (around 96%).
  • Sentence-Picture Matching: In contrast, the aphasia patients were severely impaired on this task, performing near chance levels (around 46-61%), while controls performed near ceiling (around 98%). This highlights their specific deficit in processing linguistic information.

Limitations and Future Work

  • Small Sample Size for Aphasia Patients: The study relied on only two individuals with global aphasia. While their severe impairment makes them valuable for demonstrating necessity, the findings would be strengthened by larger sample sizes or by studying individuals with more varied lesion patterns.
  • Potential for Verbal Recoding: The authors acknowledge that neurotypical participants might spontaneously "recode" visual information into a verbal format, which could explain the language network's activation. However, they argue this recoding is unlikely to be the primary or sole mechanism, especially given the task demands.
  • Future Work: The authors suggest further investigation into the nature of the language network's responses to nonverbal stimuli, exploring the role of task demands, and testing a larger sample of individuals with dissociations between language and nonverbal cognition. They also note that future work should explore whether these representations rely on sensorimotor areas or more abstract associative areas.

Summary updated

Abstract

The ability to combine individual concepts of objects, properties, and actions into complex representations of the world is often associated with language. Yet combinatorial event-level representations can also be constructed from nonverbal input, such as visual scenes. Here, we test whether the language network in the human brain is involved in and necessary for semantic processing of events presented nonverbally. In Experiment 1, we scanned participants with fMRI while they performed a semantic plausibility judgment task versus a difficult perceptual control task on sentences and line drawings that describe/depict simple agent-patient interactions. We found that the language network responded robustly during the semantic task performed on both sentences and pictures (although its response to sentences was stronger). Thus, language regions in healthy adults are engaged during a semantic task performed on pictorial depictions of events. But is this engagement necessary? In Experiment 2, we tested two individuals with global aphasia, who have sustained massive damage to perisylvian language areas and display severe language difficulties, against a group of age-matched control participants. Individuals with aphasia were severely impaired on the task of matching sentences to pictures. However, they performed close to controls in assessing the plausibility of pictorial depictions of agent-patient interactions. Overall, our results indicate that the left frontotemporal language network is recruited but not necessary for semantic processing of nonverbally presented events.
BibTeX
@inproceedings{ivanova2021language,
  author = {Anna A. Ivanova and Zachary Mineroff and Vitor Zimmerer and Nancy Kanwisher and Rosemary Varley and Evelina Fedorenko},
  title = {{The Language Network Is Recruited but Not Required for Nonverbal Event Semantics}},
  booktitle = {Neurobiology},
  year = {2021},
  doi = {10.1162/nol\_a\_00030},
}
Figure