Abstract
A wide body of evidence shows that human language processing difficulty is predicted by surprisal—a word's negative log probability in context. However, as Large Language Models (LLMs) grow in size and achieve lower perplexity, their ability to predict human reading times (RTs) paradoxically decreases. This "scaling paradox" suggests a mismatch between machine-estimated probabilities and human cognitive expectations. In this paper, we investigate whether temperature scaling can bridge this gap. We find that applying a temperature parameter ($T \approx 2.5$) to soften LLM predictions significantly improves their fit to human reading times (up to 89% improvement in $\Delta$LLH). However, our analysis reveals that this improvement is primarily driven by how models handle subword tokenization, suggesting the benefit may be more of a mathematical artifact than a reflection of human cognitive processing.
Introduction
In psycholinguistics, the surprisal of a word—its negative log probability given its context—is a robust predictor of processing difficulty, typically measured via reading times. Intuitively, a better language model should provide better estimates of surprisal. Yet, recent research has uncovered a scaling paradox: as models become more accurate at predicting text (lower perplexity), their correlation with human reading behavior often degrades.
One prominent hypothesis is that LLMs are "overconfident." Trained on massive datasets, they may assign much higher probabilities to rare words than a human would. This study tests whether re-calibrating these models using temperature scaling can improve their psycholinguistic predictive power.
Methodology
We employ temperature scaling, a post-processing technique that modifies the "logits" (raw scores) before the final softmax layer by a parameter $T$. A $T > 1$ flattens the distribution, making the model less certain and increasing the surprisal of high-probability words.
We evaluated the GPT-2 family (Small, Medium, Large, and XL) across three major reading time corpora:
- Dundee Corpus: Eye-tracking data from newspaper editorials.
- Natural Stories Corpus: Self-paced reading times for narrative texts.
- Brown Corpus: Self-paced reading times for diverse English prose.
Predictive power was quantified using delta log-likelihood ($\Delta$LLH), representing the improvement in model fit when surprisal is added to a baseline of simple predictors like word length and frequency.
Results
The Universal Optimal Temperature
Across all models and corpora, we observed a consistent peak in predictive power at a temperature of approximately $T \approx 2.5$. As shown in Figure 1, this softening of the probability distribution leads to a dramatic increase in $\Delta$LLH compared to the default $T=1$.
Applying the optimal temperature improved the predictive power by 23-43% on the Dundee corpus and a staggering 60-89% on the Natural Stories corpus. Interestingly, larger models (XL) did not necessarily require different temperatures than smaller ones, but they often showed higher absolute improvements.
The Subword "Smoking Gun"
While the fit improvements are massive, the "right reasons" for this improvement remain questionable. Our analysis shows that the benefit of temperature scaling is almost entirely concentrated in words composed of multiple subword tokens.
As illustrated in Figure 2, the $\Delta$LLH for single-token words (dotted lines) often peaks early or even declines with higher temperatures, whereas the fit for multiple-token words (dashed lines) continues to improve or remains high. This suggests that temperature scaling might be "fixing" an inherent flaw in how we aggregate subword probabilities (summing the surprisal of parts) rather than uncovering a deep truth about human cognition.
Discussion
Our findings present a nuanced view of LLM calibration. While larger LLMs are actually better calibrated to objective word probabilities than smaller ones, they remain "overconfident" relative to human expectations. However, the fact that the benefit is tied to tokenization suggests that our current methods of calculating surprisal from subword-based models are problematic.
Words split into many tokens (often rare words or named entities) have their surprisal values pushed into a more cognitively plausible range by temperature scaling. This implies that the scaling is compensating for the mathematical artifact of summing surprisals across sub-lexical units, which may not be how humans process complex words.
Conclusion
Temperature scaling is a powerful tool for improving the fit of LLM surprisal to human reading times, consistently peaking at $T \approx 2.5$. However, the discovery that this effect is driven by multi-token words suggests that researchers should be cautious in interpreting these results as a direct reflection of human "overconfidence." Future work should investigate whether these effects persist in whole-word models or if alternative methods of token-to-word surprisal aggregation can resolve the scaling paradox without manual temperature tuning.
Summary updated