Abstract
Traditional surprisal theory quantifies cognitive effort in word comprehension through contextual predictability, treating words as discrete entities. This approach fails to account for semantic or structural similarities between words. We introduce similarity-adjusted surprisal, a metric derived from Ricotta and Szeidl’s (2006) diversity index, which mathematically unifies surprisal with the concept of "information value." By incorporating graded similarities (semantic, syntactic, and orthographic), our model demonstrates improved predictive power for human reading times in naturalistic datasets, particularly when using non-contextual semantic embeddings.
Introduction: The "One-Hot" Problem in Surprisal Theory
In psycholinguistics, Surprisal Theory is the dominant framework for modeling processing effort. It suggests that the time taken to process a word is proportional to its unexpectedness in context. However, standard models typically treat every word in a vocabulary as equally distinct. For instance, if a reader expects "doctor" but encounters "physician," a standard model treats this as a complete surprise, ignoring their semantic proximity. This "one-hot" limitation overlooks the reality that processing effort is mitigated when an observed word is similar to the expected one.
Methodology
To address this, we propose a similarity-adjusted surprisal metric ($h_z$) based on the diversity index from biology. This metric is defined as the negative log of the sum of similarities between the observed word and all potential continuations, weighted by their probabilities. We evaluated three types of similarity functions ($z$):
- Semantic: Cosine similarity using GPT-2 small embeddings (both static non-contextual and dynamic contextual).
- Syntactic: Binary similarity based on Part-of-Speech (POS) tags.
- Orthographic: Similarity derived from normalized Levenshtein edit distance.
Datasets
The model was tested against human reading time (RT) data from four diverse corpora: Natural Stories (designed for complex syntax), Dundee (eye-tracking from newspaper editorials), Brown (general prose), and Provo (eye-tracking from short passages).
Results and Key Insights
The primary theoretical contribution is the proof of a monotonic relationship between similarity-adjusted surprisal and information value, reconciling two previously distinct research paths into a single framework. Empirically, the similarity-adjusted metric significantly outperformed standard surprisal in predicting reading times for the Natural Stories and Dundee datasets.
As shown in Table 1, the most substantial gains were observed in the Natural Stories corpus, where adding non-contextual semantic similarity resulted in a log-likelihood improvement (ΔL) of 0.50, a statistically significant jump in model fit. Interestingly, non-contextual semantic embeddings often provided a better fit than contextualized ones, suggesting that initial incremental reading may rely on "shallow" semantic processing rather than full contextual integration.
Discussion and Future Work
While promising, this study is limited to English and a specific set of similarity functions. Future research should explore cross-linguistic validation in morphologically rich languages, expanded similarity metrics (such as phonetic similarity), and correlation with neuroimaging data (e.g., the N400 component). Ultimately, similarity-adjusted surprisal provides a more nuanced and mathematically grounded generalization of standard surprisal theory, better reflecting the complexities of human language processing.
Summary updated