Surprisal as a Metric for Human Language Processing Effort

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2251 papers373 conceptsfrozen 2026-10-04

When we read, our brains do not process every word with equal effort. We glide over predictable words like "the" or "is" but pause when we encounter a word that is unexpected or complex. In computational psycholinguistics, this cognitive "shock" is quantified as surprisal: the negative log-probability of a word given its preceding context.

The core intuition, pioneered by John Hale in A Probabilistic Earley Parser as a Psycholinguistic Model, is that human language processing is a form of rational statistical inference. As a sentence unfolds, we maintain a probability distribution over all possible grammatical structures. When a new word arrives, we must narrow down those possibilities; the more "information" the word conveys (the more it forces us to prune our mental tree), the higher the cognitive load.

The Surprisal Theory of Difficulty

Surprisal provides a mathematical bridge between information theory and human behavior. If a word $w$ has a high probability $P(w|\text{context})$, its surprisal ($-\log P$) is low, and we process it quickly. If the word is rare or syntactically unexpected, surprisal spikes, and our reading speed slows down.

This relationship is remarkably robust. Roger Levy and colleagues demonstrated in Computational psycholinguistics that surprisal derived from simple n-gram models or parsers can accurately predict human reading times. Beyond just behavior, Stefan L. Frank showed in Word surprisal predicts N400 amplitude during reading that surprisal even correlates with the N400, a specific electrical brain response associated with processing meaning.

The Scaling Paradox

With the advent of Large Language Models (LLMs), researchers expected that better models (those with lower perplexity) would become even better predictors of human behavior. However, recent evidence suggests a "scaling paradox."

Byung-Doh Oh and William Schuler found in Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens that there is a "sweet spot" for alignment. Models trained on roughly two billion tokens predict human reading times most accurately. Beyond this point, as models become "superhuman" at predicting the next word, they actually become less like humans.

This suggests that massive modern datasets lead to a dissociation between competence (the ability to solve tasks) and psycholinguistic plausibility (the ability to mimic human processing). As noted in Large GPT-like Models are Bad Babies, scaling up models can actually make them "bad babies"—less representative of the incremental, resource-constrained way humans learn and process language.

Why Simple Surprisal Isn't Enough

While surprisal is a powerful baseline, it doesn't capture the full complexity of the human mind. Several researchers have proposed refinements to better "reverse-engineer" the reader:

Alignment and Individual Differences

A critical discovery in recent years is that the way we "align" AI models affects their cognitive realism. Tatsuki Kuribayashi and Yohei Oseki found that instruction tuning—the process of making LLMs more helpful—actually degrades their ability to predict human reading times (Psychometric Predictive Power of Large Language Models). The base models, which focus purely on next-word prediction, remain the most "human-like" in their internal reflexes.

Furthermore, LLMs do not predict a single "average" human. Research into individual differences suggests that models are better at predicting the behavior of readers with lower verbal intelligence or working memory. High-capacity readers appear to be less sensitive to surprisal, as they can more efficiently integrate unexpected information into their mental models.

By treating LLMs as "computational subjects" and comparing their surprisal values to human eye-tracking and neural data, researchers are moving toward a unified theory of how both biological and artificial systems navigate the statistical landscape of language.

Papers in this version

Showing 60 of 251 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 343 papers
  2. Version 1 · 2026-10-04 · 251 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_surprisal_based_cognitive_modeling_f131bccd,
  title = {surprisal-based cognitive modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surprisal-based-cognitive-modeling/rdr_f131bccd59c3432ba0468a2ae44cf6f4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_surprisal_as_a_metric_for_human_language_f131bccd_v1,
  title = {Surprisal as a Metric for Human Language Processing Effort},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surprisal-as-a-metric-for-human-language-processing-effort/rdr_f131bccd59c3432ba0468a2ae44cf6f4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}