Introduction
Surprisal Theory has long been a cornerstone of psycholinguistics, positing that the cognitive effort required to process a word is proportional to its negative log-probability in context: $h(w) = -\log p(w|context)$. However, this traditional formulation treats words as discrete, unrelated entities. For instance, it assumes that encountering an unexpected but semantically related word (e.g., "cat" instead of "dog") incurs the same processing cost as an entirely unrelated word (e.g., "refrigerator"), provided their raw probabilities are identical. This "categorical blind spot" fails to account for the nuanced semantic and syntactic relationships that influence human language processing.
Theoretical Framework: Similarity-adjusted Surprisal
To address this limitation, Meister et al. (2024) introduce similarity-adjusted surprisal ($h_z$). This metric is formally derived from diversity indices used in ecology, specifically the Ricotta and Szeidl (2006) index, which measures biodiversity by accounting for the relatedness of species.
The proposed metric calculates an "expected similarity" for a word $w_t$ given its context: $$h_z(w_t) = -\log \sum_{w' \in V} z(w_t, w') p(w' | w_{<t})$$ where $z(w_t, w')$ represents a similarity function between 0 and 1. This formulation provides a mathematical bridge to "information value," showing a monotonic relationship between the two concepts and reconciling previously disparate theories of linguistic predictability.
Methodology
The authors evaluated the predictive power of $h_z$ using four distinct similarity functions:
- Semantic: Using GPT-2 small embeddings to calculate cosine similarity (both contextual and non-contextual).
- Syntactic: Based on shared Part-of-Speech (POS) tags.
- Orthographic: Based on character-level edit distance.
These metrics were tested against human reading time (RT) data from four major corpora: Brown (self-paced reading), Dundee (eye-tracking), Natural Stories (complex syntax), and Provo (short stimuli).
Results and Discussion
The experimental results demonstrate that similarity-adjusted surprisal consistently improves the prediction of human reading times over standard surprisal models, particularly in complex linguistic contexts.
As shown in Table 1, the most significant gains were observed in the Natural Stories and Dundee datasets, which feature longer and more complex discourse. Notably, non-contextual semantic similarity often outperformed contextualized versions. This suggests that initial cognitive effort during incremental reading may be driven by "shallow" semantic processing—the immediate recognition of core word meanings—rather than the full integration of contextual nuances.
Furthermore, the results highlight the complementarity of the new metric. Similarity-adjusted surprisal does not replace Shannon surprisal; instead, it captures unique variance related to how humans navigate "communicative equivalences." As the similarity function $z$ becomes stricter (approaching an identity matrix), $h_z$ mathematically converges back to standard Shannon surprisal.
Conclusion
This work provides a robust formal extension to Surprisal Theory, grounding it in the mathematics of diversity indices. By accounting for word similarity, the model better reflects the complexities of human language processing. Future research will focus on extending these findings to non-English languages and exploring other physiological markers of cognitive load, such as the N400 brain response.
Reference: Meister, C., Giulianelli, M., & Pimentel, T. (2024). Towards a Similarity-adjusted Surprisal Theory. Proceedings of EMNLP 2024.
Summary updated