A resource-rational model of human processing of recursive linguistic structure

Michael Hahn, Richard Futrell, Roger Lévy, Edward GibsonProceedings of the National Academy of Sciences 2022 · DOI

TL;DR

This paper introduces a novel "resource-rational" model that unifies two classic theories of language processing: expectation-based and memory-based. By modeling memory as a rational trade-off between precision and computational cost, the model makes fine-grained predictions about sentence processing difficulty, particularly for complex nested structures, which are then validated in three behavioral experiments.

What Problem Does This Paper Address?

Understanding why some sentences are harder to process than others is a central goal in psycholinguistics. Two main theories have emerged: expectation-based models, which suggest difficulty arises from low predictability of words, and memory-based models, which attribute difficulty to limitations in working memory capacity. While both have empirical support, they often make conflicting predictions, especially for complex linguistic structures like nested clauses (e.g., "The report that the doctor who the lawyer distrusted annoyed the patient was surprising"). Standard theories struggle to explain why certain variations in these structures, beyond just the number of nested clauses, impact processing difficulty. This paper aims to bridge this gap by proposing a unified theoretical framework that accounts for both expectation and memory constraints.

Methods

The core of the paper is a "resource-rational lossy-context surprisal" model. This model posits that human language processing is "resource-rational," meaning it optimizes the use of limited cognitive resources. Specifically, it models memory representations as being imperfect ("lossy") but rationally optimized to minimize the expected downstream processing effort. This involves a trade-off: more precise memory representations are costly, while less precise ones are cheaper but can lead to errors.

The model is implemented using contemporary machine learning techniques, specifically a neural network that learns "retention probabilities" for words in the context. These probabilities determine which words are kept in memory and which are "erased," based on factors like recency and word frequency. The model then uses Bayesian inference to reconstruct the most likely context and predict the next word. Processing difficulty is then measured by the "surprisal" (unpredictability) of the next word given this reconstructed context.

The authors scale this model by training it on large text corpora, allowing it to learn the statistical regularities of language. This scaled-up implementation enables the model to make detailed, fine-grained predictions for arbitrary sentences.

Datasets

  • English Wikipedia: Used for training the model's retention probabilities and estimating the statistical structure of English.
  • English Web Treebank, AnCoRA treebank (Spanish), HDT Treebank (German): Used to identify nouns suitable for creating experimental stimuli.
  • English Wikipedia (2.3 billion words), German Wikipedia (800 million words), Spanish Wikipedia (500 million words): Used to estimate "embedding bias" for nouns (how likely they are to be followed by a "that"-clause).
  • Experimental data: Collected from three behavioral experiments involving native speakers of English, Spanish, and German.

Key Insights and Ideas

  • Memory is Rationally "Lossy": The paper argues that human memory isn't just limited; it's rationally limited. We don't store everything perfectly because it's computationally expensive. Instead, we store what's most useful, accepting some loss of information.
  • Unifying Expectations and Memory: The model demonstrates how a single framework can explain phenomena previously attributed solely to either expectation or memory. The "lossy" nature of memory means that even predictable words can be hard to process if the relevant context has been "forgotten."
  • Statistical Cues Matter More Than Previously Thought: The model predicts, and experiments confirm, that subtle statistical properties of words (like "embedding bias") significantly influence processing difficulty, even in complex nested structures. This goes beyond simple predictability or structural complexity.
  • Semantic Compatibility Plays a Role: The compatibility of verbs with nouns in nested clauses also affects processing difficulty, a factor not well-accounted for by previous models.

Key Results

The model's predictions were tested and confirmed in three experiments:

  • Experiment 1 (Statistical Cues):
    • Reading times for the final verb in nested sentences were higher with more embedding levels (THREE > TWO), as predicted by memory-based theories.
  • Crucially, reading times were modulated by the "embedding bias" of the initial noun. For simple sentences (ONE condition), higher embedding bias led to easier processing. For nested sentences (TWO/THREE conditions), higher embedding bias led to harder processing. This pattern is a unique prediction of the resource-rational model and not explained by existing theories.
  • Experiment 2 (Semantic Cues):
    • Replicated the findings from Experiment 1.
  • Added semantic compatibility: When the second-to-last verb was incompatible with the first noun (e.g., "report... cured"), processing the final verb was easier than when it was compatible (e.g., "report... annoyed"). This is because incompatibility disfavors certain non-veridical interpretations, reinforcing the correct structure.
  • Experiment 3 (Production Study):
    • In sentence completion tasks across English, Spanish, and German, participants were more likely to produce grammatically complete sentences (i.e., with the correct number of verbs) when the initial noun had a high embedding bias. This suggests that statistical cues influence not just comprehension but also production expectations.

Limitations and Future Work

  • Free Parameter: The model has a free parameter, $\delta$, representing the average number of retained context words. While the authors explored a range of values, fitting this parameter to individual subjects and relating it to individual differences is suggested for future work.
  • Simplification of Memory: The model currently treats the recent context as a sequence of words. Future work could explore more sophisticated memory representations that capture meaning beyond individual words, potentially using advanced machine learning techniques.
  • Similarity-Based Interference: The authors note that their current framework does not explicitly account for "similarity-based interference" (e.g., when two similar phrases compete for processing resources), which is an interesting avenue for future research.
  • Generalizability: The authors suggest that similar resource-rational principles might apply to processing in other complex domains beyond language, which warrants further investigation.

Summary updated

Abstract

A major goal of psycholinguistic theory is to account for the cognitive constraints limiting the speed and ease of language comprehension and production. Wide-ranging evidence demonstrates a key role for linguistic expectations: A word's predictability, as measured by the information-theoretic quantity of surprisal, is a major determinant of processing difficulty. But surprisal, under standard theories, fails to predict the difficulty profile of an important class of linguistic patterns: the nested hierarchical structures made possible by recursion in human language. These nested structures are better accounted for by psycholinguistic theories of constrained working memory capacity. However, progress on theory unifying expectation-based and memory-based accounts has been limited. Here we present a unified theory of a rational trade-off between precision of memory representations with ease of prediction, a scaled-up computational implementation using contemporary machine learning methods, and experimental evidence in support of the theory's distinctive predictions. We show that the theory makes nuanced and distinctive predictions for difficulty patterns in nested recursive structures predicted by neither expectation-based nor memory-based theories alone. These predictions are confirmed 1) in two language comprehension experiments in English, and 2) in sentence completions in English, Spanish, and German. More generally, our framework offers computationally explicit theory and methods for understanding how memory constraints and prediction interact in human language comprehension and production.
BibTeX
@inproceedings{hahn2022resource,
  author = {Michael Hahn and Richard Futrell and Roger Lévy and Edward Gibson},
  title = {{A resource-rational model of human processing of recursive linguistic structure}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2022},
  doi = {10.1073/pnas.2122602119},
}
Figure