Abstract
Large Language Models (LLMs) are often "too confident" compared to humans, making them surprisingly poor at predicting human reading behavior. By applying temperature scaling to "soften" these probability estimates, researchers achieved up to an 89% improvement in modeling human reading times. This study discovers that the benefit is primarily driven by how models handle rare words and multi-token sequences, revealing a fundamental trade-off between objective model calibration and human-like processing.
Introduction
In psycholinguistics, "surprisal" (defined as the negative log probability of a word given its context) is the gold-standard metric for predicting human processing difficulty. The core intuition is simple: the more surprising a word is, the longer a human takes to read it. Consequently, one might expect that "better" language models—those with lower perplexity—would provide superior surprisal estimates and thus better predict reading times.
However, recent research has uncovered a "scaling paradox": as LLMs grow larger and more accurate at predicting text, their ability to model human reading behavior actually decreases. This paper investigates the hypothesis that this gap is a calibration issue. LLMs are exposed to orders of magnitude more data than any human, leading to "overconfident" predictions, especially for rare words or named entities. The authors explore whether temperature scaling can bridge this gap by softening model probabilities to better match human-like expectations.
Methods
The researchers employed temperature scaling, a post-processing technique typically used to improve model calibration. Before the final softmax layer, the model's logits are divided by a temperature parameter $T$. When $T > 1$, the resulting probability distribution is "softened": high-probability predictions become less certain, and low-probability predictions become more likely.
The study evaluated four variants of GPT-2 (small, medium, large, and xl) across three major human reading time corpora:
- Dundee Corpus: Eye-tracking data of subjects reading newspaper editorials.
- Natural Stories Corpus: Self-paced reading times for narrative texts.
- Brown Corpus: Self-paced reading times across various English genres.
To measure predictive power, the authors used Linear Mixed-Effects (LME) models. The primary metric was Delta Log-Likelihood ($\Delta llh$), quantifying the improvement in fit to human data when scaled surprisal is included as a predictor compared to a baseline.
Key Results
The Optimal Human Temperature
Temperature scaling led to a systematic and massive improvement in predicting reading times across all corpora. On the Natural Stories corpus, the improvement in $\Delta llh$ reached as high as 89%. Across different models and datasets, the optimal temperature ($T^*$) for matching human reading times consistently fell between 2.0 and 3.0, averaging around 2.5.
As shown in Figure 2, while smaller models (like GPT-2 small) generally outperformed larger ones at predicting reading times at $T=1$, temperature scaling significantly reduced this performance gap.
The Role of Subword Tokenization
A key insight of the study is that the improvement from temperature scaling is not uniform. It is almost entirely driven by words that the model splits into multiple subword tokens (e.g., rare words or complex names).
Figure 4 demonstrates that for single-token words, the benefit of scaling is significantly lower. Named entities, which are often multi-token, showed the highest sensitivity. Models are often "too sure" about these specific tokens, whereas humans find them consistently difficult to process. Temperature scaling corrects this by increasing the surprisal of these rare sequences to levels that better align with human effort.
Discussion and Conclusion
The results highlight a fundamental trade-off: Calibration vs. Human-Likeness. Improving a model's fit to human reading times (making it more "human-like") significantly worsens its objective calibration. Larger LLMs are actually better calibrated to objective word frequencies in the real world, but this very accuracy makes them less like humans, who have limited memory and exposure.
Limitations and Future Work
The authors acknowledge that the benefit might be partially an artifact of subword tokenization. If a word is split into multiple tokens, the model sums their surprisals, often creating an estimate that diverges from human "whole-word" processing. Future work should test these findings on models with whole-word vocabularies. Additionally, while the optimal $T$ was found via grid search, developing automated methods to estimate $T$ based on corpus characteristics remains an open challenge.
Summary updated