Modeling Brain Activity Associated with Pronoun Resolution in English and Chinese

Jixing Li, Murielle Fabre, Wen-Ming Luh, John HaleNAACL 2018

TL;DR

This paper uses the ACT-R cognitive architecture to model pronoun resolution in English and Chinese, hypothesizing that Chinese speakers rely more on salience due to fewer grammatical cues. By correlating model predictions with fMRI data, they found distinct brain activation patterns, suggesting that language-specific features influence the cognitive and neural mechanisms of pronoun processing.

What Problem Does This Paper Address?

Understanding how humans resolve pronouns (e.g., knowing who "he" or "she" refers to) is a fundamental challenge in linguistics and cognitive science. While English relies on morphosyntactic cues like gender and number agreement, Chinese pronouns often lack these markers, leading to the hypothesis that Chinese speakers might depend more heavily on the "salience" or prominence of potential antecedents in the discourse. This paper aims to test this hypothesis by modeling pronoun resolution using the ACT-R cognitive architecture and examining its correlation with brain activity in English and Chinese speakers. The core question is whether differences in language structure lead to observable differences in cognitive processing and brain activation during pronoun resolution.

Methods

The researchers employed a computational modeling approach using the ACT-R (Adaptive Control of Thought-Rational) cognitive architecture. Specifically, they adapted the ACT-R model for pronoun resolution, which calculates an "activation level" for potential antecedents based on factors like recency, frequency, and grammatical role. This activation level is interpreted as a measure of salience.

To test their hypothesis, they performed two main analyses:

  1. Model Performance: They compared the ACT-R model's predicted difficulty of pronoun resolution (inversely related to activation level) with actual brain activity.
  2. fMRI Correlation: They correlated the model's "difficulty" measure with brain activity (BOLD signals) recorded via fMRI while participants listened to audiobooks in English and Chinese. This allowed them to see if the model's prediction of cognitive effort aligned with neural responses.

They also conducted a simpler GLM analysis correlating brain activity directly with the presence of pronouns, serving as a baseline comparison.

Datasets

The study used audiobook versions of Antoine de Saint-Exupéry's "The Little Prince" in both English and Chinese translation. The texts were annotated for pronouns and non-pronominal mentions.

  • English Audiobook: Contained 1755 pronouns (excluding possessives, reflexives, and dummy pronouns) and 3127 non-pronominal entities.
  • Chinese Audiobook: Contained 1785 pronouns (excluding possessives and reflexives) and 2947 non-pronominal mentions.

The data was processed and annotated using the brat annotation tool.

Key Insights and Ideas

  • Language-Specific Salience Reliance: The study provides evidence that Chinese speakers, lacking explicit morphosyntactic cues for pronoun resolution, rely more heavily on the salience of antecedents compared to English speakers. This is supported by both model performance and distinct brain activation patterns.
  • ACT-R as a Cognitive Model: The ACT-R model's difficulty measure (derived from activation levels) successfully correlated with brain activity, suggesting it captures a meaningful aspect of the cognitive effort involved in pronoun resolution.
  • Dissociable Neural Networks: The research highlights that different languages engage distinct neural networks for pronoun resolution. English processing, particularly when dealing with syntactic complexity, seems to involve areas like Broca's area, while Chinese processing, especially when salience is paramount, engages regions like the Angular Gyrus.
  • Gender Information's Neural Footprint: The study suggests that the presence of grammatical gender information in English influences brain activation, potentially involving areas associated with gender processing, whereas Chinese, lacking this, might rely more on general salience mechanisms.

Key Results

  • Model Accuracy: The ACT-R model showed higher overall accuracy for pronoun resolution in Chinese (71%) compared to English (66%). This difference was more pronounced for 2nd and 3rd person pronouns.
  • Activation Levels: The ACT-R model predicted significantly higher activation levels for Chinese antecedents compared to English antecedents, supporting the salience hypothesis.
  • fMRI Findings:
    • For English speakers, increased ACT-R difficulty for 3rd person pronouns was associated with activity in the left Middle Temporal Gyrus (MTG), left Inferior Frontal Gyrus (IFG), and left Superior Frontal Gyrus (SFG).
  • For Chinese speakers, increased ACT-R difficulty was associated with activity in the left Angular Gyrus, left SFG, and left MTG.
  • Direct comparison revealed stronger activation in the right Angular Gyrus, right MTG, and right Precuneus for English speakers, and stronger activation in the left Angular Gyrus for Chinese speakers when considering ACT-R difficulty.

Limitations and Future Work

  • Speech vs. Reading: The study focused solely on auditory stimuli (audiobooks). The authors acknowledge that conclusions might differ for reading comprehension, as Chinese pronouns can have different forms in written text.
  • Future Directions: The authors suggest future work should compare brain activity during pronoun resolution in both reading and listening comprehension. They also propose exploring how cross-linguistic differences in pronoun resolution might be further investigated through increased cross-fertilization between computational linguistics and cognitive neuroscience.

Summary updated

BibTeX
@inproceedings{li2018modeling,
  author = {Jixing Li and Murielle Fabre and Wen-Ming Luh and John Hale},
  title = {{Modeling Brain Activity Associated with Pronoun Resolution in English and Chinese}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure