TL;DR
This paper introduces a novel character-based language model integrated into a structural parser that significantly improves predictions of human reading times compared to traditional word-level models. The findings suggest that a deeper understanding of morphology, phonotactics, and orthography is crucial for accurately modeling sentence processing.
What Problem Does This Paper Address?
Understanding how humans process language in real-time is a fundamental goal in psycholinguistics. A key concept is "surprisal," which quantifies how unexpected a word is in a given context. Higher surprisal generally correlates with slower reading times and increased cognitive load. While large language models (LLMs) have become popular for estimating surprisal, they primarily operate at the word level. This leaves open the question of whether incorporating finer-grained linguistic information, such as character-level features, within a more human-like incremental processing model could lead to better predictions of human reading behavior. Existing work has shown that LLMs, despite their scale, don't always perfectly align with human processing costs, hinting at a gap in their representational capacity for psycholinguistic phenomena.
Methods
The core of this paper is a character-based word generation model integrated into a structural parser-based processing model.
- Structural Parser: The model uses a left-corner parser, a type of incremental parser that mimics human sentence processing by making decisions about lexical attachments and grammatical structures step-by-step. This parser maintains a "store state" representing the current parsing context.
- Character-Based Word Model: This is the novel contribution. Instead of just predicting the next word, it models how a word is generated from a lemma (the base form of a word) and a morphological rule. This allows the model to capture morphological information (like prefixes and suffixes), phonotactics (sound patterns), and orthographic complexity (spelling patterns). It uses recurrent neural networks (RNNs) to process character sequences and learn embeddings for syntactic categories and predicate contexts.
- Surprisal Estimation: The model calculates word generation probabilities based on the parser's state and the character-based word model. Surprisal is then derived as the negative log probability of the predicted word.
Datasets
- Natural Stories Corpus: Used for self-paced reading time experiments. This corpus contains 10 naturalistic stories read by 181 subjects.
- Dundee Corpus: Used for eye-tracking experiments. This corpus consists of gaze durations from 10 subjects reading 20 newspaper editorials.
- Natural Stories Corpus (fMRI data): Used for fMRI experiments. This data comes from 78 subjects listening to recorded stories.
The paper also references and compares against surprisal estimates derived from various large-scale pre-trained language models (GLSTMSurp, JLSTMSur, GPT2Surp) and other parser-based models (RNNGSurp, vSLCSurp, JLCSurp), which were trained on datasets like the Penn Treebank and Gigaword.
Key Insights and Ideas
- Character-level detail matters for human processing: The study strongly suggests that models incorporating character-level information (morphology, phonotactics, orthography) provide a more accurate account of human sentence processing than large word-level language models.
- Parser-based models offer a human-like advantage: Integrating character models into an incremental, structural parser framework yields better predictions of human reading times than simply using large LLMs. This implies that the structure of processing, not just the prediction of the next word, is critical.
- Surprisal isn't just about word prediction: The findings challenge the assumption of a monotonic relationship between test perplexity (a measure of LLM performance) and predictive power for human behavior. A model with higher perplexity on test data but better character-level integration can outperform LLMs in predicting human reading times.
- Morphology and orthography are more important than previously thought: The success of the character-based model highlights the significant role these linguistic features play in how humans understand sentences.
Key Results
- Self-paced Reading: The character-based structural model (CharWSurp) provided substantially better fits to self-paced reading data compared to large language models and a baseline word-level parser. The improvement was statistically significant (p < 0.0001).
- Eye-tracking: Similar to reading times, CharWSurp significantly improved the fit for eye-tracking (go-past durations) data, outperforming other models.
- fMRI: Interestingly, for fMRI data, CharWSurp did not contribute as much as some LSTM-based language models. This suggests that different measures of cognitive processing (latency vs. neural activity) might capture different aspects of surprisal.
- Out-of-Vocabulary (OOV) words: The character-based approach inherently handles OOV words better, and excluding OOV words from other models did not significantly alter the main trends, reinforcing the value of the character-level approach.
Limitations and Future Work
- fMRI Discrepancy: The paper notes that the character-based model's contribution was less pronounced for fMRI data compared to latency-based measures. This difference suggests that future work could explore how different neural signals capture various facets of processing difficulty and how character-level information might be represented in brain activity.
- Generalizability: While the study shows broad improvement across categories, further investigation into how the model generalizes across different linguistic domains and tasks could be beneficial.
- Model Complexity: The paper presents a complex model. Future work could focus on streamlining or optimizing the character-based integration for broader adoption.
Summary updated