Language beyond the language system: Dorsal visuospatial pathways support processing of demonstratives and spatial language during naturalistic fast fMRI

Roberta Rocca, Kenny R. Coventry, Kristian Tylén, Marlene Staib, Torben Ellegaard Lund, Mikkel Wallentin

TL;DR

This study uses naturalistic fMRI to investigate how the brain processes spatial demonstratives (like "this" and "that"). It reveals that understanding these words relies on a network of brain regions involved in both language and visuospatial processing, particularly the dorsal "where" pathway, suggesting a shared neural architecture for spatial language and cognition.

What Problem Does This Paper Address?

Understanding language, especially semantically underspecified expressions like demonstratives ("this one"), requires more than just processing words. It involves integrating linguistic information with visual cues, attention, and pragmatic inference. This paper specifically tackles how the brain achieves this integration for spatial demonstratives, which are crucial for establishing joint attention and referring to objects in our environment. While previous research has hinted at the involvement of visuospatial pathways in language, this study aims to provide direct neuroimaging evidence for the role of the dorsal "where" stream in processing spatial language.

Methods

The researchers employed a naturalistic fMRI paradigm where 28 participants listened to a specially crafted dialogue. This dialogue was designed to include a controlled number of spatial demonstratives. To capture brain activity at a fine temporal resolution (word-level), they used a fast multiband-EPI acquisition sequence (TR = 388ms) combined with finite impulse response (FIR) modeling of the hemodynamic response. They also conducted multivariate similarity analyses to compare brain responses to demonstratives with responses to "wh-" words (what, where, who) to probe the distinction between semantic and spatial processing.

Datasets

Not explicitly detailed as datasets for training or evaluation in the traditional machine learning sense. The study used a custom-created, naturalistic dialogue as the stimulus material.

Key Insights and Ideas

  • Language and Vision Share Neural Resources: The findings strongly suggest that processing spatial language, particularly demonstratives, recruits brain regions typically associated with non-linguistic visuospatial processing. This challenges a strict modular view of language processing.
  • The Dorsal "Where" Stream is Key for Spatial Language: The study provides compelling evidence that the dorsal visual stream, known for processing spatial information ("where" pathway), is not only involved in visual spatial tasks but also plays a significant role in understanding linguistic expressions of space.
  • Demonstratives are a Bridge: Spatial demonstratives act as a crucial interface, linking linguistic meaning with real-world spatial context, attention, and perception. Their processing engages a distributed network rather than isolated language-specific areas.
  • No Clear Proximal/Distal Segregation: Contrary to predictions, the study did not find distinct brain regions for processing proximal ("here") versus distal ("there") demonstratives, suggesting that the brain might not have a hard-wired segregation for object reachability in this context.

Key Results

  • Spatial demonstratives bilaterally activated parietal areas (supramarginal gyrus, angular gyrus, precuneus) involved in information integration and visuospatial processing.
  • Frontal regions, including the right frontal eye field (FEF), were recruited for attentional orienting and reference frame shifts.
  • Multivariate similarity analyses showed that brain responses to spatial demonstratives were more similar to responses to the "where" wh-word than to "what" or "who," supporting the involvement of the dorsal "where" stream in spatial language.
  • While no significant segregation was found for proximal vs. distal demonstratives, some magnitude differences were observed, with distal demonstratives showing slightly higher activation in certain areas.

Limitations and Future Work

  • No Eye-Tracking Data: The study acknowledges that the observed activation in the frontal eye fields (FEF) could be influenced by actual eye movements, which were not recorded. Future studies should include eye-tracking to disentangle these effects.
  • Exploratory Contrast for Proximal/Distal: The post-hoc analysis comparing proximal and distal demonstratives used a lenient threshold, and the small effect size means these findings should be considered pointers for future research rather than definitive evidence.
  • Controlled Context for Proximal/Distal: The linguistic context for proximal and distal demonstrative forms was not fully controlled in the text, which could have influenced the results.
  • Future Directions: The authors suggest further experiments to directly test hypotheses about processing demands for proximal vs. distal demonstratives and to explore the functional specialization within the dorsal stream for different types of spatial reference frames. They also highlight the need for further investigation into the observed asymmetry in frontal eye field activation.

Summary updated

Abstract

Abstract Spatial demonstratives are powerful linguistic tools used to establish joint attention. Identifying the meaning of semantically underspecified expressions like “this one” hinges on the integration of linguistic and visual cues, attentional orienting and pragmatic inference. This synergy between language and extralinguistic cognition is pivotal to language comprehension in general, but especially prominent in demonstratives. In this study, we aimed to elucidate which neural architectures enable this intertwining between language and extralinguistic cognition using a naturalistic fMRI paradigm. In our experiment, 28 participants listened to a specially crafted dialogical narrative with a controlled number of spatial demonstratives. A fast multiband-EPI acquisition sequence (TR = 388ms) combined with finite impulse response (FIR) modelling of the hemodynamic response was used to capture signal changes at word-level resolution. We found that spatial demonstratives bilaterally engage a network of parietal areas, including the supramarginal gyrus, the angular gyrus, and precuneus, implicated in information integration and visuospatial processing. Moreover, demonstratives recruit frontal regions, including the right FEF, implicated in attentional orienting and reference frames shifts. Finally, using multivariate similarity analyses, we provide evidence for a general involvement of the dorsal (“where”) stream in the processing of spatial expressions, as opposed to ventral pathways encoding object semantics. Overall, our results suggest that language processing relies on a distributed architecture, recruiting neural resources for perception, attention, and extra-linguistic aspects of cognition in a dynamic and context-dependent fashion.
BibTeX
@article{rocca2020language,
  author = {Roberta Rocca and Kenny R. Coventry and Kristian Tylén and Marlene Staib and Torben Ellegaard Lund and Mikkel Wallentin},
  title = {{Language beyond the language system: Dorsal visuospatial pathways support processing of demonstratives and spatial language during naturalistic fast fMRI}},
  journal = {NeuroImage},
  year = {2020},
  doi = {10.1016/j.neuroimage.2019.116128},
}

Loading related content...

Figure