The Depth of a Prediction: Where Language Models Resolve a Word Predicts Human Reading Time

Donato Riccio, Fabrizio MaturoarXiv 2026 · arXiv

TL;DR

Standard NLP models predict human reading effort based on how "surprising" a word is at the final output layer. This paper shows that where a model settles on that prediction—its "resolution depth"—is just as important: words that take more layers to resolve correlate with longer human reading times, even when the final prediction is identical.

What Problem Does This Paper Address?

Psycholinguistics has long used surprisal theory to explain human reading behavior: the harder a word is to predict (higher surprisal), the longer a reader pauses on it. Typically, researchers calculate this surprisal using the final output of a Large Language Model (LLM).

However, LLMs are deep Transformers that refine their internal "guesses" across dozens of layers. A word might be easy to guess by layer 2, or it might remain a mystery until layer 40. The authors ask: does the internal "effort" or "timing" of the model’s prediction process mirror the cognitive effort of a human reader? If two words are equally surprising at the end, but one was "obvious" early on while the other required deep processing, do humans treat them differently?

Methods

The authors introduce a new metric called Resolution Depth ($D$).

  1. Layerwise Readouts: Using techniques like the "tuned lens," they extract the model's prediction for a word at every single layer, not just the last one.
  2. Surprisal Trajectories: This creates a "trajectory" of surprisal across the network's depth.
  3. Calculating $D$: Resolution depth is the "center of gravity" of change in this trajectory. A low $D$ means the model figured out the word in the early layers; a high $D$ means the prediction changed significantly in the final layers.
Figure 1: Mean layerwise surprisal trajectories (tuned lens) of OneStop words in the middle fifth of final sur- prisal (40th to 60th percentile) under GPT-2 XL, split into the fifth with the lowest and the fifth with the high- est resolution depth D (Natural Stories with Pythia 2.8B in Figure 4a). Dotted lines mark each group’s mean D; n is the number of analyzed words in the band. Both groups end at the same surprisal; the late-resolved words keep a high surprisal through more of the network.

The researchers then tested whether $D$ could predict human reading times (from eye-tracking and self-paced reading data) better than standard baselines, controlling for factors like word length, frequency, and final-layer surprisal.

Datasets

The study is impressively broad, evaluating ten different models (GPT-2, OPT, and Pythia families) across eight English reading corpora:

  • Eye-tracking: Provo, GECO, OneStop, MECO, ZuCo, and UCL eye-tracking.
  • Self-paced reading: Natural Stories and UCL self-paced. In total, they analyzed over 81,000 words.

Key Insights and Ideas

  • The "Aha!" Moment Matters: The most striking finding is that humans read words more slowly if the model resolves them late. This suggests that the "computational depth" required by a model captures something about the linguistic complexity that also taxes human brains.
  • Beyond Final Surprisal: $D$ provides information that final-layer surprisal and entropy miss. It’s not just about what the model predicts, but how long it takes to get there.
  • Function Words are "Deep": Interestingly, the words that models resolve late are often common function words (like and, in, the, to) when they appear at the start of a new phrase. In these cases, the local context is ambiguous, and the model (and likely the human) needs more global context to settle on the right structure.
  • Model-Specific "Effort": The effect is present in trained models but disappears in untrained ones, proving that this isn't just an artifact of network architecture—it's a property of learned linguistic processing.

Key Results

  • Improved Prediction: Adding resolution depth to the model improved the prediction of held-out human reading times by 0.26 millinats per word ($p = 0.001$). For context, this is about a third of the gain provided by contextual entropy, a well-established predictor.
  • Consistency: The effect was positive in seven out of eight corpora when compared against a "shuffled" baseline (where layer order is randomized).
  • Magnitude: For certain models and datasets, a one-standard-deviation increase in $D$ corresponded to a 1.1% to 1.6% increase in "go-past" reading time (the time spent on a word before moving forward).
Figure 3: Held-out gain from resolution depth on the primary measure for each corpus and model, in millinats per word. Asterisks mark one-sided permutation p < 0.05 for the corpus and model alone.

Limitations and Future Work

  • Effect Size: While statistically significant, the absolute gain in predictive power is relatively small.
  • Corpus Heterogeneity: The strength of the effect varied significantly between datasets. For example, the gain was very strong in the OneStop and Natural Stories corpora but near zero in others like GECO. The authors suggest that different text types or reader populations might influence how much resolution depth matters.
  • English-Only: The study was limited to English. Future work could explore if these "depth" effects are universal across languages with different morphological or syntactic structures.
  • Better Lenses: The results depend on the "lens" used to peek into the model. The authors note that more advanced probing techniques might reveal even stronger links between model depth and human cognition.

Summary updated

Abstract

Surprisal from a language model's final layer predicts how long people read a word, yet the model computes its prediction across many layers. We read each word's prediction at every layer with the tuned lens and summarize this layerwise surprisal trajectory by its resolution depth, the depth around which its change is centered. On eight English eye-tracking and self-paced reading corpora (81,957 words) and ten GPT-2, OPT and Pythia models, words that a model resolves later, and the words after them, are read more slowly than equally surprising words. Resolution depth improves held-out prediction beyond final-layer surprisal, entropy and the best single layer by 0.26 millinats per word (pooled p = 0.001), about a third of entropy's gain, and each model family gains, GPT-2 and Pythia through one corpus each. The net gain varies between corpora (random-effects mean 0.07, p = 0.37); against layer-shuffled depth, which adds as many coefficients, it is positive in seven of eight corpora (random-effects mean 0.23, p = 0.016), and untrained networks show no gain under the logit lens. Pooled over corpora, the effect holds with corrected word probabilities and beyond first-to-last-layer divergences, and a whole-trajectory functional regression adds no significant information beyond resolution depth.
BibTeX
@misc{riccio2026depth,
  author = {Donato Riccio and Fabrizio Maturo},
  title = {{The Depth of a Prediction: Where Language Models Resolve a Word Predicts Human Reading Time}},
  year = {2026},
  eprint = {2611.25970},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25970},
}
Figure