PIHKers at CMCL 2021 Shared Task: Cosine Similarity and Surprisal to Predict Human Reading Patterns.

Lavinia Salicchi, Alessandro LenciNAACL 2021

TL;DR

This paper explores how to predict human reading patterns (specifically eye-tracking data) using computational linguistic features. By combining cosine similarity (measuring semantic coherence) and surprisal (measuring predictability) from advanced language models, the authors achieved significant improvements over baseline methods, offering a more nuanced understanding of reading comprehension.

What Problem Does This Paper Address?

Understanding how humans process language is a fundamental goal in cognitive science and linguistics. Eye-tracking studies provide rich data on reading behavior, revealing that semantic coherence and predictability of words heavily influence how we read. However, predicting these fine-grained eye-tracking patterns from linguistic features has been a challenging task. This paper addresses the need for computational models that can effectively capture these psycholinguistic influences, aiming to predict eye-tracking metrics like fixation duration and reading time from textual input. This has implications for building more sophisticated natural language understanding systems and generating new hypotheses about cognitive processes.

Methods

The authors propose a two-step approach to predict eye-tracking features. First, they leverage pre-trained word embedding models (like BERT, ELMo, GloVe) to extract two key linguistic features:

  • Cosine Similarity: This measures the semantic relatedness between a target word and its preceding context. They experimented with various methods to compute the context vector, including summing word embeddings.
  • Surprisal: This quantifies how unexpected a word is given the preceding context, typically derived from language models. They specifically used BERT, GPT2-xl, and Neural Complexity models for surprisal computation.

To select the most informative features, they performed a preliminary investigation by calculating Spearman's correlation between these linguistic features and the eye-tracking data. The features with the highest correlation were then used in regression models.

For the regression task, they compared nine different models from Scikit-learn (including Gradient Boosting Regressor, Random Forest Regressor, and Linear Regression). The primary evaluation metric was Mean Absolute Error (MAE) on the eye-tracking features. They tested three main configurations: surprisal only, cosine similarity only, and a combination of both.

Datasets

The primary dataset used in this study is ZuCo (Hollenstein et al., 2018). ZuCo contains EEG and eye-tracking data from 12 English speakers reading natural texts. For the CMCL 2021 shared task, 800 sentences were used for training and approximately 200 sentences for testing.

Additionally, the authors used GECO (Cop et al., 2017) and Provo (Luke and Christianson, 2018) for preliminary feature selection. GECO is a corpus of Agatha Christie novels with eye-tracking data from 33 subjects, while Provo consists of 55 short English texts with eye-tracking data from 85 subjects. These datasets were used to extract mean reading times, fixation durations, and number of fixations for words, helping to identify features that correlate well with human reading behavior.

Key Insights and Ideas

  • Synergy of Semantic Coherence and Predictability: The most significant finding is that combining cosine similarity (semantic coherence) and surprisal (predictability) leads to substantially better predictions of eye-tracking data than using either feature alone. This highlights that both aspects of language processing are crucial and offer complementary information.
  • BERT and GPT2-xl Shine: Advanced contextualized embeddings, particularly BERT (for cosine similarity) and GPT2-xl (for surprisal), proved to be the most effective in capturing the linguistic features that predict reading patterns. This underscores the power of large language models in encoding nuanced linguistic information relevant to human cognition.
  • Feature Selection is Key: The preliminary step of selecting features based on their correlation with eye-tracking data was vital. This ensured that the regression models were trained on the most predictive linguistic signals, rather than a broad set of potentially noisy features.
  • Gradient Boosting Regressor Dominates: Among the tested regression models, the Gradient Boosting Regressor (GBR) consistently yielded the best performance when combined with the selected linguistic features.

Key Results

The authors' model significantly outperformed the baseline. For the shared task, their system achieved an average MAE of 4.3877 across eye-tracking features, compared to a baseline MAE of 7.3699. This represents a substantial improvement. The best model in the competition achieved an MAE of 3.8134, and their model's performance was only slightly higher (0.5743 MAE difference).

Specifically, they found that:

  • BERT layer 22 CLS context embeddings provided the best cosine similarity features (mean correlation of -0.62).
  • GPT2-xl provided the best surprisal features (mean correlation of 0.40).
  • The combination of BERT cosine similarity and GPT2-xl surprisal, along with word length and word frequency, fed into a Gradient Boosting Regressor, yielded the best results for most eye-tracking features.

Limitations and Future Work

  • fixProp Anomaly: The model struggled to accurately predict fixProp (the proportion of participants fixating a word), achieving a significantly higher MAE for this feature. The authors hypothesize that fixProp might be influenced by a complex interplay of factors not fully captured by their current features and suggest exploring new regressors and textual features for this specific metric.
  • Model Improvement Potential: The authors suggest exploring new types of regressors and word embeddings, as well as incorporating additional textual features like sentence length and information about preceding words, to further enhance model performance.
  • Interplay of Features: While their results suggest that cosine similarity and surprisal convey distinct and useful information, they acknowledge that there might be some overlap or interplay between these measures, as suggested by Frank (2017). Further research could delve deeper into this relationship.

Summary updated

BibTeX
@inproceedings{salicchi2021pihkers,
  author = {Lavinia Salicchi and Alessandro Lenci},
  title = {{PIHKers at CMCL 2021 Shared Task: Cosine Similarity and Surprisal to Predict Human Reading Patterns.}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure