A model of local coherence effects in human sentence processing as consequences of updates from bottom-up prior to posterior beliefs

Klinton Bicknell, Roger LevyNAACL 2009 · OpenReview

TL;DR

This paper proposes a novel computational model for "local coherence effects" in human sentence processing, explaining them as a consequence of rational belief updates rather than processing errors. By framing sentence comprehension as a process of updating probabilistic beliefs about sentence structure, the model successfully predicts experimental findings that challenge previous rational processing theories.

What Problem Does This Paper Address?

Human sentence processing is a complex task where we incrementally build a hierarchical understanding of a sentence from a linear stream of words. While many processing phenomena have been successfully explained by assuming comprehenders rationally use probabilistic information from various sources (like word frequencies and syntactic structures), a class of effects called "local coherence effects" has remained puzzling. These effects suggest that comprehenders sometimes get "stuck" on locally plausible interpretations, even when global sentence context makes them unlikely. This seems to contradict the idea of rational processing, where all available information should be used. The paper aims to reconcile local coherence effects with a rational processing framework by proposing a specific mechanism for how probabilistic beliefs are updated during comprehension.

Methods

The core of the paper's contribution is a new model that conceptualizes sentence processing as a belief-update process. Specifically, it models how a comprehender's initial "bottom-up" beliefs about the likely syntactic structures for a given string of words are updated to "posterior" beliefs that incorporate the broader sentence context.

The key idea is that larger belief updates are more processing-intensive. The model quantifies this update using the Kullback-Leibler (KL) divergence between the prior and posterior probability distributions over syntactic structures. A large KL divergence signifies a substantial shift in beliefs, which is predicted to correspond to higher processing difficulty.

To calculate these prior and posterior probabilities, the authors leverage a probabilistic Earley parser. This parser, trained on a stochastic context-free grammar (SCFG), provides the necessary probabilities to estimate the belief distributions. The model then uses these probabilities to predict the summed KL divergence for different parts of a sentence, with higher divergence indicating predicted processing difficulty.

Datasets

  • Brown Corpus: Used to estimate the rule probabilities for the stochastic context-free grammar (SCFG) and lexical rewrite probabilities for part-of-speech tags.
  • Tabor et al. (2004) Experiment 1 Materials: The paper uses a subset of these materials (12 out of 20 items) for empirical evaluation. These materials were specifically designed to elicit local coherence effects, involving ambiguous verbs and relative clause structures.

Key Insights and Ideas

  • Local Coherence as Rational Belief Update: The paper reframes local coherence effects not as a failure of rationality, but as a consequence of the computational cost associated with updating probabilistic beliefs. When the bottom-up interpretation strongly conflicts with the context-informed posterior, a large belief update is required, leading to processing difficulty.
  • The Role of Priors: The model highlights the importance of "prior" beliefs – the initial probabilistic expectations about sentence structure based solely on the words encountered so far. These priors can be misleading if they strongly favor an interpretation that is later contradicted by the global context.
  • Quantifying Processing Load: Using KL divergence as a measure of belief update provides a principled, quantitative way to predict processing difficulty. This moves beyond qualitative descriptions of parsing difficulty.
  • Connection to Surprisal: The belief-update framework is linked to surprisal-based theories of sentence processing, where processing difficulty is related to the unexpectedness of incoming information. Here, the "surprisal" is framed in terms of the effort needed to reconcile initial expectations with new contextual information.
  • Potential for Precomputation: The authors speculate that the need for rapid access to prior probabilities might suggest that comprehenders precompute these beliefs for common short sequences of words, rather than computing them from scratch each time.

Key Results

The implemented model, using a small SCFG trained on the Brown Corpus, successfully predicted the experimental results from Tabor et al. (2004). Specifically:

  • The model predicted increased processing difficulty (higher summed KL divergence) at the critical words in locally coherent regions, particularly in ambiguous conditions (e.g., "the player tossed a frisbee").
  • The model captured the super-additive difficulty observed in the ambiguous reduced condition (1a) of Tabor et al.'s experiment, where the effect was greater than the sum of individual effects of ambiguity and reduction. This was attributed to the strong prior belief in a simple, locally coherent parse.
  • The predicted difficulty was localized to specific words and substrings, aligning with the experimental findings for different substring lengths (1, 2, and 3 words).

Limitations and Future Work

  • Grammar Size: The current model uses a small, hand-crafted grammar. A significant future direction is to scale up the model to use a broad-coverage grammar, allowing it to make predictions for a wider range of sentences and linguistic phenomena.
  • Correlations in Priors: The model acknowledges potential correlations between different syntactic categories in its prior and posterior vectors (e.g., if a ROOT category begins, so do S, S-base, and NP-base). Future work could explore dimensionality reduction techniques to handle these correlations.
  • Approximation of Priors: While the model computes priors from grammar probabilities, the authors suggest that humans might approximate these priors through frequency-based learning from large corpora, rather than direct grammatical computation. Testing this hypothesis by manipulating substring frequencies is a proposed future experiment.
  • Exhaustive Evaluation: The authors suggest more exhaustive evaluation on a variety of datasets is needed.

Summary updated

Abstract

Human sentence processing involves integrating probabilistic knowledge from a variety of sources in order to incrementally determine the hierarchical structure for the serial input stream. While a large number of sentence processing effects have been explained in terms of comprehenders' rational use of probabilistic information, effects of local coherences have not. We present here a new model of local coherences, viewing them as resulting from a belief-update process, and show that the relevant probabilities in our model are calculable from a probabilistic Earley parser. Finally, we demonstrate empirically that an implemented version of the model makes the correct predictions for the materials from the original experiment demonstrating local coherence effects.
BibTeX
@inproceedings{bicknell2009model,
  author = {Klinton Bicknell and Roger Levy},
  title = {{A model of local coherence effects in human sentence processing as consequences of updates from bottom-up prior to posterior beliefs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
  url = {https://openreview.net/forum?id=HJWRuXb_-H},
}
Figure