Towards a Similarity-adjusted Surprisal Theory

Clara Meister, Mario Giulianelli, Tiago Pimentel

Introduction

Surprisal Theory has long been a cornerstone of psycholinguistics, positing that the cognitive effort required to process a word is proportional to its negative log-probability in context: $h(w) = -\log p(w|context)$. However, this traditional formulation treats words as discrete, unrelated entities. For instance, it assumes that encountering an unexpected but semantically related word (e.g., "cat" instead of "dog") incurs the same processing cost as an entirely unrelated word (e.g., "refrigerator"), provided their raw probabilities are identical. This "categorical blind spot" fails to account for the nuanced semantic and syntactic relationships that influence human language processing.

Theoretical Framework: Similarity-adjusted Surprisal

To address this limitation, Meister et al. (2024) introduce similarity-adjusted surprisal ($h_z$). This metric is formally derived from diversity indices used in ecology, specifically the Ricotta and Szeidl (2006) index, which measures biodiversity by accounting for the relatedness of species.

The proposed metric calculates an "expected similarity" for a word $w_t$ given its context: $$h_z(w_t) = -\log \sum_{w' \in V} z(w_t, w') p(w' | w_{<t})$$ where $z(w_t, w')$ represents a similarity function between 0 and 1. This formulation provides a mathematical bridge to "information value," showing a monotonic relationship between the two concepts and reconciling previously disparate theories of linguistic predictability.

Methodology

The authors evaluated the predictive power of $h_z$ using four distinct similarity functions:

  • Semantic: Using GPT-2 small embeddings to calculate cosine similarity (both contextual and non-contextual).
  • Syntactic: Based on shared Part-of-Speech (POS) tags.
  • Orthographic: Based on character-level edit distance.

These metrics were tested against human reading time (RT) data from four major corpora: Brown (self-paced reading), Dundee (eye-tracking), Natural Stories (complex syntax), and Provo (short stimuli).

Results and Discussion

The experimental results demonstrate that similarity-adjusted surprisal consistently improves the prediction of human reading times over standard surprisal models, particularly in complex linguistic contexts.

Table 1: Predictive improvement ($\Delta L$) of similarity-adjusted surprisal and information value across different similarity functions and datasets.

As shown in Table 1, the most significant gains were observed in the Natural Stories and Dundee datasets, which feature longer and more complex discourse. Notably, non-contextual semantic similarity often outperformed contextualized versions. This suggests that initial cognitive effort during incremental reading may be driven by "shallow" semantic processing—the immediate recognition of core word meanings—rather than the full integration of contextual nuances.

Furthermore, the results highlight the complementarity of the new metric. Similarity-adjusted surprisal does not replace Shannon surprisal; instead, it captures unique variance related to how humans navigate "communicative equivalences." As the similarity function $z$ becomes stricter (approaching an identity matrix), $h_z$ mathematically converges back to standard Shannon surprisal.

Conclusion

This work provides a robust formal extension to Surprisal Theory, grounding it in the mathematics of diversity indices. By accounting for word similarity, the model better reflects the complexities of human language processing. Future research will focus on extending these findings to non-English languages and exploring other physiological markers of cognitive load, such as the N400 brain response.

Reference: Meister, C., Giulianelli, M., & Pimentel, T. (2024). Towards a Similarity-adjusted Surprisal Theory. Proceedings of EMNLP 2024.

Summary updated

Abstract

Surprisal theory posits that the cognitive effort required to comprehend a word is determined by its contextual predictability, quantified assurprisal. Traditionally, surprisal theory treats words as distinct entities, overlooking any potential similarity between them. Giulianelli et al. (2023) address this limitation by introducing information value, a measure of predictability designed to account for similarities between communicative units. Our work leverages Ricotta and Szeidl’s (2006) diversity index to extend surprisal into a metric that we term similarity-adjusted surprisal, exposing a mathematical relationship between surprisal and information value. Similarity-adjusted surprisal aligns with information value when considering graded similarities and reduces to standard surprisal when words are treated as distinct. Experimental results with reading time data indicate that similarity-adjusted surprisal adds predictive power beyond standard surprisal for certain datasets, suggesting it serves as a complementary measure of comprehension effort.
BibTeX
@inproceedings{meister2024towards,
  author = {Clara Meister and Mario Giulianelli and Tiago Pimentel},
  title = {{Towards a Similarity-adjusted Surprisal Theory}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.921},
}
Figure