Contributions of Propositional Content and Syntactic Category Information in Sentence Processing

Byung-Doh Oh, William SchulerNAACL 2021

TL;DR

This paper introduces a novel computational model that integrates both the meaning (propositional content) and the grammatical structure (syntactic category) of sentences to predict human processing difficulty. By carefully ablating these components, the authors demonstrate that both meaning and grammar play distinct and significant roles in how quickly and easily we comprehend language, with syntactic information being particularly crucial for predicting reading times.

What Problem Does This Paper Address?

Understanding how humans process language in real-time is a fundamental question in cognitive science and computational linguistics. While it's widely accepted that our brains anticipate upcoming words based on context, the exact nature of this prediction remains debated. A prominent theory suggests that "surprisal" – how unexpected a word is given the preceding context – drives processing difficulty. However, existing computational models often struggle to capture what kind of information humans actually use to form these predictions. Specifically, it's unclear how much of this prediction relies on the meaning of the sentence versus its grammatical structure. This paper aims to disentangle these two crucial aspects of language processing.

Methods

The core of this work is an incremental left-corner parser. This type of parser builds a representation of a sentence word by word, making decisions about how to attach each new word both lexically (based on its meaning and grammatical role) and grammatically (based on the overall sentence structure).

What's novel here is how they incorporate propositional content. They augment the parser with "predicate context vectors" derived from a generalized categorial grammar. These vectors represent the meaning associated with predicates (like verbs) and their arguments, capturing semantic relationships.

To isolate the contributions of meaning and grammar, they create ablated models:

  • Full Model: Incorporates both propositional content and syntactic categories.
  • Content-Ablated Model: Uses only syntactic categories, ignoring propositional content.
  • Category-Ablated Model: Uses only propositional content, ignoring syntactic categories.

These models are trained to predict word probabilities, and the resulting "surprisal" values are then used to predict human reading times and eye-gaze data.

Datasets

  • Training: The models were trained on a reannotation of sections 02-21 of the Wall Street Journal (WSJ) corpus using a generalized categorial grammar.
  • Evaluation (Linguistic Accuracy): The trained parsers were evaluated on sections 22-23 of the WSJ corpus and the Natural Stories Corpus.
  • Evaluation (Behavioral Data):
    • Self-paced reading times: Data from the Natural Stories Corpus (Futrell et al., 2018) was used.
  • Eye-tracking data: Data from the Dundee Corpus (Kennedy et al., 2003) was used.

Key Insights and Ideas

  • Meaning and Grammar are Distinct Predictors: The paper strongly suggests that humans don't just rely on one type of information. Both the semantic "gist" of a sentence and its grammatical scaffolding are actively used during processing.
  • Syntactic Categories Matter Significantly for Reading Time: While both components contribute, the ablated models reveal that syntactic information has a particularly strong and consistent impact on predicting how quickly people read sentences.
  • Propositional Content is More Evident in Eye-Gaze: Interestingly, propositional content seems to have a more pronounced effect on eye-gaze durations compared to self-paced reading times, suggesting these measures might capture slightly different aspects of processing difficulty.
  • Ablation is Key to Understanding Contributions: The ability to cleanly "turn off" either propositional content or syntactic category information in their model is crucial for demonstrating their independent and combined effects.

Key Results

  • Linguistic Accuracy: The full model performed comparably to a previous state-of-the-art parser on the WSJ corpus. Notably, the content-sensitive model generalized better to the more diverse Natural Stories Corpus.
  • Self-paced Reading Times:
    • The category-ablated model (using only syntactic info) showed a statistically significant contribution to predicting reading times over the content-ablated model in 8 out of 9 tests.
  • The content-ablated model (using only propositional info) did not show a significant contribution over the category-ablated model.
  • Eye-tracking Data (Go-past durations):
    • The content-ablated model showed a statistically significant contribution over the category-ablated model in all 9 tests.
  • The category-ablated model showed a significant contribution over the content-ablated model in 3 out of 9 tests.
  • Filler-Gap Constructions: In complex sentences where an object is extracted early and needs to be integrated with a later verb, the full model's surprisal significantly improved predictions of reading times over the content-ablated model, suggesting propositional content plays a key role in resolving these difficult constructions.

Limitations and Future Work

  • Convergence Issues: The authors note that regression models sometimes failed to converge, particularly when using raw surprisal predictors instead of difference predictors. This is a common challenge in complex statistical modeling.
  • Spillover Variants: While they used "spillover" variants of predictors (accounting for the delay in processing), they acknowledge that including too many can lead to identifiability issues.
  • Future Work: The paper suggests further exploration of how these models can be used to examine differential effects of propositional content in memory-intensive constructions, as hinted at in their filler-gap experiment. They also imply that further refinement of how propositional content is represented could yield even richer insights.

Summary updated

BibTeX
@inproceedings{oh2021contributions,
  author = {Byung-Doh Oh and William Schuler},
  title = {{Contributions of Propositional Content and Syntactic Category Information in Sentence Processing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure