TL;DR
This paper investigates how language models (LMs) predict human reading times by incorporating individual cognitive differences. They found that LMs' ability to predict reading times improves when individual cognitive scores are considered, and that LMs tend to emulate readers with lower verbal intelligence, suggesting potential biases in their predictions for high-intelligence readers.
What Problem Does This Paper Address?
Traditionally, studies on how word predictability (measured by surprisal and entropy) affects reading effort have focused on group averages. This approach overlooks the significant role individual cognitive differences play in language processing. For instance, people with higher working memory or verbal intelligence might process language differently. This paper aims to bridge this gap by exploring how individual cognitive capacities influence the relationship between LM-derived predictability measures and actual human reading times, and whether LMs themselves exhibit biases towards certain cognitive profiles.
Methods
The researchers used linear-mixed models (LMMs) to predict reading times. They extracted surprisal and contextual entropy from five pre-trained generative LMs (GPT-2 base/large, Llama 2 7B/13B, Mixtral). These predictability measures, along with word-level predictors like lexical frequency and word length, were used to predict first-pass reading times (FPRT). The core of their analysis involved:
- Baseline analysis: Assessing the predictive power of surprisal and entropy alone.
- Interaction analysis (H1): Investigating if including interactions between predictability measures and individual psychometric scores improves prediction accuracy.
- Coefficient analysis (H2): Examining the direction and magnitude of these interactions to understand how cognitive scores modulate the effects of surprisal/entropy.
- Group-based analysis (H3): Splitting participants into high- and low-performing groups based on their psychometric scores to see if LMs predict reading times better for one group over the other.
They operationalized predictive power as the change in log-likelihood (ΔLL) of the model.
Datasets
The study utilized the Individual Differences Corpus (InDiCo). This corpus contains:
- Eye-tracking-while-reading and self-paced-reading data from 61 native German speakers.
- A comprehensive battery of psychometric scores covering four cognitive domains: cognitive control, working memory, intelligence, and reading fluency.
The InDiCo corpus is notable for its rich psychometric data, allowing for a detailed examination of individual differences in language processing.
Key Insights and Ideas
- Cognitive Capacities Matter: Individual cognitive differences are not just noise; they significantly enhance the predictive power of LM-derived surprisal and entropy for human reading times. This suggests that a more nuanced understanding of language processing requires accounting for individual variation.
- LMs Emulate Specific Profiles: LMs don't universally model all readers. The study provides evidence that the tested LMs tend to emulate readers with lower verbal intelligence, meaning their predictability estimates might be less accurate for individuals with higher verbal intelligence.
- Reading Fluency is Key: Interactions between reading fluency scores and predictability measures (especially surprisal) showed particularly high predictive power, suggesting that fluent readers might rely less on predictive processing or are better at integrating unexpected information.
- Working Memory's Complex Role: While high working memory capacity generally correlated with lower surprisal effects, the results are complex and warrant further investigation, especially in light of previous ERP studies suggesting higher working memory individuals might be more prone to garden-path effects.
Key Results
- Improved Prediction: Incorporating interactions between predictability measures (surprisal/entropy) and cognitive scores significantly improved the prediction of reading times across most psychometric tests (H1 supported).
- Sensitivity Varies: Individuals with higher cognitive scores generally showed lower sensitivity to predictability effects (surprisal/entropy), meaning their reading times were less affected by word predictability. The Simon test (non-verbal inhibitory control) was a notable exception, where higher scores were associated with larger surprisal effects.
- LM Biases: LMs consistently showed higher predictive power for individuals with low verbal intelligence scores (H3 supported). This was observed across all tested models and for both surprisal and entropy measures.
- Model Size vs. Predictability: For surprisal, increasing LM size led to decreasing predictive power in interactions with cognitive scores. For entropy, this trend was less consistent. GPT-2 base showed the highest predictive power for interactions.
Limitations and Future Work
- Median Split: Splitting participants into high/low performers based on the median might not create perfectly homogeneous groups, especially for individuals with scores close to the median. Future work could explore more sophisticated clustering approaches.
- Test-Retest Reliability: Individual surprisal effects can vary day-to-day due to factors like wakefulness. While the InDiCo dataset mitigates this by combining data from multiple sessions, it remains a potential limitation.
- Generalizability: The InDiCo corpus, while diverse, primarily consists of participants with academic backgrounds. Future research could explore these effects in more diverse populations.
- Model Choice: The choice of LM significantly impacts the resulting predictability measures and their predictive power. This highlights the need for careful model selection when applying LM-derived metrics in psycholinguistic research or NLP applications.
- LM-Produced Text: The paper focuses on LM biases in predicting human processing. Future work could investigate whether text produced by LMs is biased towards being processed more easily by individuals with specific cognitive characteristics, which could inform tasks like text summarization or simplification.
Summary updated