Real-Time Parallel Processing of Grammatical Structure in the Fronto-Striatal System: A Recurrent Network Simulation Study Using Reservoir Computing

Xavier Hinaut, Peter Ford DomineyPLoS 2013 · DOI

TL;DR

This paper proposes a novel computational model that simulates how the human brain processes sentence structure in real-time. By leveraging reservoir computing, the model demonstrates how the fronto-striatal system can predict and update sentence meaning as words arrive, offering a potential explanation for neurophysiological signals like the P600 ERP.

What Problem Does This Paper Address?

Understanding how humans process language in real-time, especially grammatical structure, is a significant challenge. Neurophysiological studies, particularly Event-Related Potentials (ERPs), show that the brain anticipates upcoming words and grammatical structures, with signals like the P600 ERP reflecting processing complexity and violations. The fronto-striatal system (frontal cortex and striatum) is implicated in this process, but the precise mechanisms remain unclear. This paper aims to bridge this gap by proposing a computational model that explains how this real-time grammatical processing and prediction might occur within this neural system.

Methods

The authors employ a reservoir computing (RC) framework to model sentence processing. In this approach:

  • Recurrent Neural Network (Reservoir): A fixed recurrent neural network, analogous to the prefrontal cortex (specifically BA47), acts as the "reservoir." This network processes the sequential input of words and encodes the temporal structure of the sentence.
  • Readout Layer: A separate layer, analogous to the striatum, acts as the "readout." This layer learns to decode the activity patterns from the reservoir to produce the desired output, which represents the "coded meaning" of the sentence (i.e., the thematic roles of nouns and verbs).
  • Plastic Connections: The connections between the reservoir and the readout layer are learnable, mimicking synaptic plasticity in the cortico-striatal system.
  • Reservoir Computing Advantage: RC allows for efficient training of the readout layer using linear regression, enabling the model to learn complex temporal dependencies without needing to train the entire recurrent network. This is a significant advantage over traditional recurrent network training.

Datasets

The study utilizes several corpora of grammatical constructions:

  • Initial Corpus: 26 distinct grammatical constructions (based on Dominey et al., 2006).
  • Extended Corpus I: 462 novel grammatical constructions generated from a context-free grammar, varying in complexity and structure.
  • Extended Corpus II: A much larger corpus of over 90,000 constructions, designed to explore scaling capabilities and introduce properties like redundancy and ambiguity.
  • Scrambled Corpus: Used for control experiments, this corpus had the word order of the 462 constructions randomized to test generalization without inherent grammatical structure.

Key Insights and Ideas

  • Cortico-Striatal Homology: The paper proposes a compelling analogy between the cortico-striatal system and the reservoir computing framework, where the cortex acts as the fixed reservoir and the striatum as the plastic readout. This provides a neurobiologically plausible computational substrate for sentence processing.
  • Real-Time Prediction and Reanalysis: The model demonstrates that predictions about sentence meaning are made on-line as words are processed. Crucially, these predictions can be confirmed or updated in real-time as new words arrive, mirroring human sentence reanalysis.
  • P600 ERP as a "Processing Cost": The authors suggest that the changes in readout activity (analogous to striatal activity) in response to unexpected or less frequent grammatical structures could be a computational correlate of the P600 ERP. This "change in neural activity" can be interpreted as a measure of "processing cost" or reanalysis effort.
  • Corpus Frequency Matters: The model's responses are sensitive to the statistical properties of the training corpus. Less frequent grammatical constructions lead to larger changes in readout activity, aligning with findings that such constructions elicit larger P600 responses in humans.
  • Generalization Beyond Training: The model shows a significant ability to generalize to novel grammatical constructions not explicitly present in the training set, suggesting it learns underlying grammatical structure rather than just memorizing examples.

Key Results

  • Syntactic Comprehension: The model successfully learned to determine thematic roles for a set of 26 grammatical constructions.
  • On-line Reanalysis: The model exhibited on-line reanalysis of sentence meaning, with changes in readout activity reflecting updates to predictions as words were processed (e.g., distinguishing between subject- and object-relative clauses).
  • P600-like Activity: Changes in readout activity, particularly in response to less frequent constructions, were proposed as a neural correlate of the P600 ERP.
  • Grammatical Generalization: The model demonstrated generalization to novel constructions, achieving reasonable error rates (e.g., ~15% sentence error in cross-validation with the 462 corpus). Performance degraded significantly when the grammatical structure was removed (scrambled corpus), highlighting the importance of learned structure.
  • Scaling and Robustness: With larger corpora (up to 90,000 constructions), the model continued to show generalization capabilities, with performance improving with increased training data. The system also showed robustness to variations in parameters and input timing.
  • Discourse Processing: The model could integrate information from preceding sentences to resolve anaphoric references in subsequent sentences, demonstrating an ability to maintain and utilize discourse context.

Limitations and Future Work

  • "Toy" Demonstrations: While promising, the authors acknowledge that some of the initial corpora and experiments represent "toy" demonstrations, and further exploration with more extended and complex corpora is needed.
  • Ambiguity: The presence of ambiguous constructions in larger corpora (up to 12%) limits the achievable sentence error rate, suggesting a ceiling effect.
  • P600 Analogy: The analogy between the model's readout activity changes and the human P600 ERP is tentative. While it captures some key aspects (frequency effects, reanalysis), the precise neurophysiological origin of the P600 (potentially involving striatal influence on cortex) is complex and requires further investigation.
  • Future Work: The authors suggest exploring more extended discourse situations, investigating the role of different neural substrates (e.g., purely cortico-cortical networks), and further validating the model against more detailed neurophysiological data. They also highlight the potential of reservoir computing for modeling other cortico-striatal functions.

Summary updated

Abstract

Sentence processing takes place in real-time. Previous words in the sentence can influence the processing of the current word in the timescale of hundreds of milliseconds. Recent neurophysiological studies in humans suggest that the fronto-striatal system (frontal cortex, and striatum--the major input locus of the basal ganglia) plays a crucial role in this process. The current research provides a possible explanation of how certain aspects of this real-time processing can occur, based on the dynamics of recurrent cortical networks, and plasticity in the cortico-striatal system. We simulate prefrontal area BA47 as a recurrent network that receives on-line input about word categories during sentence processing, with plastic connections between cortex and striatum. We exploit the homology between the cortico-striatal system and reservoir computing, where recurrent frontal cortical networks are the reservoir, and plastic cortico-striatal synapses are the readout. The system is trained on sentence-meaning pairs, where meaning is coded as activation in the striatum corresponding to the roles that different nouns and verbs play in the sentences. The model learns an extended set of grammatical constructions, and demonstrates the ability to generalize to novel constructions. It demonstrates how early in the sentence, a parallel set of predictions are made concerning the meaning, which are then confirmed or updated as the processing of the input sentence proceeds. It demonstrates how on-line responses to words are influenced by previous words in the sentence, and by previous sentences in the discourse, providing new insight into the neurophysiology of the P600 ERP scalp response to grammatical complexity. This demonstrates that a recurrent neural network can decode grammatical structure from sentences in real-time in order to generate a predictive representation of the meaning of the sentences. This can provide insight into the underlying mechanisms of human cortico-striatal function in sentence processing.
BibTeX
@inproceedings{hinaut2013real,
  author = {Xavier Hinaut and Peter Ford Dominey},
  title = {{Real-Time Parallel Processing of Grammatical Structure in the Fronto-Striatal System: A Recurrent Network Simulation Study Using Reservoir Computing}},
  booktitle = {PLoS},
  year = {2013},
  doi = {10.1371/journal.pone.0052946},
}
Figure