TL;DR
Standard NLP models predict human reading effort based on how "surprising" a word is at the final output layer. This paper shows that where a model settles on that prediction—its "resolution depth"—is just as important: words that take more layers to resolve correlate with longer human reading times, even when the final prediction is identical.
What Problem Does This Paper Address?
Psycholinguistics has long used surprisal theory to explain human reading behavior: the harder a word is to predict (higher surprisal), the longer a reader pauses on it. Typically, researchers calculate this surprisal using the final output of a Large Language Model (LLM).
However, LLMs are deep Transformers that refine their internal "guesses" across dozens of layers. A word might be easy to guess by layer 2, or it might remain a mystery until layer 40. The authors ask: does the internal "effort" or "timing" of the model’s prediction process mirror the cognitive effort of a human reader? If two words are equally surprising at the end, but one was "obvious" early on while the other required deep processing, do humans treat them differently?
Methods
The authors introduce a new metric called Resolution Depth ($D$).
- Layerwise Readouts: Using techniques like the "tuned lens," they extract the model's prediction for a word at every single layer, not just the last one.
- Surprisal Trajectories: This creates a "trajectory" of surprisal across the network's depth.
- Calculating $D$: Resolution depth is the "center of gravity" of change in this trajectory. A low $D$ means the model figured out the word in the early layers; a high $D$ means the prediction changed significantly in the final layers.
The researchers then tested whether $D$ could predict human reading times (from eye-tracking and self-paced reading data) better than standard baselines, controlling for factors like word length, frequency, and final-layer surprisal.
Datasets
The study is impressively broad, evaluating ten different models (GPT-2, OPT, and Pythia families) across eight English reading corpora:
- Eye-tracking: Provo, GECO, OneStop, MECO, ZuCo, and UCL eye-tracking.
- Self-paced reading: Natural Stories and UCL self-paced. In total, they analyzed over 81,000 words.
Key Insights and Ideas
- The "Aha!" Moment Matters: The most striking finding is that humans read words more slowly if the model resolves them late. This suggests that the "computational depth" required by a model captures something about the linguistic complexity that also taxes human brains.
- Beyond Final Surprisal: $D$ provides information that final-layer surprisal and entropy miss. It’s not just about what the model predicts, but how long it takes to get there.
- Function Words are "Deep": Interestingly, the words that models resolve late are often common function words (like and, in, the, to) when they appear at the start of a new phrase. In these cases, the local context is ambiguous, and the model (and likely the human) needs more global context to settle on the right structure.
- Model-Specific "Effort": The effect is present in trained models but disappears in untrained ones, proving that this isn't just an artifact of network architecture—it's a property of learned linguistic processing.
Key Results
- Improved Prediction: Adding resolution depth to the model improved the prediction of held-out human reading times by 0.26 millinats per word ($p = 0.001$). For context, this is about a third of the gain provided by contextual entropy, a well-established predictor.
- Consistency: The effect was positive in seven out of eight corpora when compared against a "shuffled" baseline (where layer order is randomized).
- Magnitude: For certain models and datasets, a one-standard-deviation increase in $D$ corresponded to a 1.1% to 1.6% increase in "go-past" reading time (the time spent on a word before moving forward).
Limitations and Future Work
- Effect Size: While statistically significant, the absolute gain in predictive power is relatively small.
- Corpus Heterogeneity: The strength of the effect varied significantly between datasets. For example, the gain was very strong in the OneStop and Natural Stories corpora but near zero in others like GECO. The authors suggest that different text types or reader populations might influence how much resolution depth matters.
- English-Only: The study was limited to English. Future work could explore if these "depth" effects are universal across languages with different morphological or syntactic structures.
- Better Lenses: The results depend on the "lens" used to peek into the model. The authors note that more advanced probing techniques might reveal even stronger links between model depth and human cognition.
Summary updated