CogNLP-Sheffield at CMCL 2021 Shared Task: Blending Cognitively Inspired Features with Transformer-based Language Models for Predicting Eye Tracking Patterns

Peter Vickers, Rosa Wainwright, Harish Tayyar Madabushi, Aline VillavicencioNAACL 2021

TL;DR

This paper explores how to predict human eye-tracking patterns during reading using language models. The researchers found that a smaller Transformer model (XLNet-base) surprisingly outperformed a larger one, and that while some cognitively inspired features like word length were helpful, multi-word expression features offered little benefit. This work contributes to understanding how language models can capture human cognitive processes.

What Problem Does This Paper Address?

Predicting human eye-tracking patterns during reading is a challenging task that offers insights into cognitive processes like language acquisition and comprehension. Traditionally, researchers have relied on linguistic features to model reading behavior. However, recent advancements in large language models (LLMs), particularly Transformer-based ones, have shown remarkable capabilities in understanding language. This paper addresses the question of whether these powerful LLMs, when combined with cognitively inspired linguistic features, can more accurately predict eye-tracking behavior than either approach alone. Specifically, it investigates if LLMs implicitly capture cognitive information and if explicitly providing such information enhances their predictive power for eye-tracking data.

Methods

The researchers employed two main approaches:

  • Feature Engineering: They extracted a rich set of cognitively and linguistically inspired features. These included:
  • Linguistic Features: Part-of-speech tags, word length, word frequency (raw and Zipf), concreteness norms, and indicators for sentence start/end.
  • Reading-Specific Features: Raw word lengths and categorical representations of word length relative to average saccade length. They also experimented with features from adjacent words (previous and next).
  • GECO Corpus Features: Summary statistics (mean, median, standard deviation) of gaze features from the GECO corpus.
  • Multi-word Expression (MWE) Features: Indicator features for MWE presence, categorical features for MWE syntactic patterns, and Skip-gram embeddings for MWEs, along with compositionality scores.
  • Transformer-based Language Models: They utilized XLNet, a generalized autoregressive pre-training method, as their primary LLM.
  • Model 1 (Feature Rich): This model combined the engineered linguistic features with word embeddings from XLNet. A 3-layer dense neural network was used to predict the target gaze features.
  • Model 2 (XLNet): This model primarily relied on XLNet's contextual embeddings. The final hidden representation of the first sub-word token for each word was used, concatenated with word length. This was then fed into a 3-layer dense neural network.

Both models were trained using a Huber loss objective and optimized with the AdamW optimizer. For the Feature Rich model, a Random Forest Regressor was also explored.

Datasets

  • CMCL 2021 Shared Task Data: This dataset, derived from the Zurich Cognitive Language Processing Corpora (Hollenstein et al., 2018, 2020), consists of 991 sentences (800 training, 191 test) with token-level gaze features. The gaze features include First Fixation Duration (FFD), Total Reading Time (TRT), Gaze Past Time (GPT), Number of Fixations (nFix), and Fixation Proportion (fixProp).
  • GECO Corpus: Used for generating type-level summary statistics of gaze features.
  • Wikitext-103 Corpus: Used for generating MWE lexicon and Skip-gram word embeddings.

Key Insights and Ideas

  • Smaller is Sometimes Better: The XLNet-base model surprisingly outperformed the larger XLNet-large model. This is hypothesized to be due to the limited task-specific training data, potentially causing the larger model to underfit.
  • LLM Embeddings are Powerful: XLNet embeddings alone (Model 2) were highly effective, subsuming information from most other features, especially for predicting nFix. This suggests LLMs implicitly capture a significant amount of linguistic and cognitive information.
  • Cognitive Features' Mixed Success: While word length proved to be a consistently useful feature, even for the XLNet model, other cognitively inspired features, particularly those related to Multi-word Expressions (MWEs), provided little to no benefit. This was surprising given prior research suggesting MWEs have distinct processing advantages.
  • Pre-training is Crucial: Randomizing the weights of the XLNet model (removing pre-training) significantly increased the Mean Absolute Error (MAE), highlighting the importance of the model's extensive pre-training.
  • GECO Features as Proxies: The Feature Rich model made substantial use of GECO features, suggesting they might provide valuable approximations for gaze distributions that are hard to learn directly from the limited task data.

Key Results

The primary evaluation metric was Mean Absolute Error (MAE).

  • Model 2 (XLNet-base) achieved an MAE of 3.914 on the validation splits, ranking 5th overall in the CMCL Shared Task.
  • Model 1 (Feature Rich) achieved an MAE of 4.007, ranking 7th overall.
  • A combined model (ElasticNet with XLNet + ALL Features) achieved a slightly better MAE of 3.912, but the authors focused their analysis on the two distinct models.
  • Removing XLNet's pre-trained weights increased the MAE to 4.675, demonstrating the impact of pre-training.

Limitations and Future Work

  • MWE Representation: The authors acknowledge that the lack of benefit from MWE features might not mean MWEs are unimportant, but rather that their current representation might not be optimal for capturing their cognitive relevance. Further research into MWE representations is suggested.
  • Model Size vs. Data: The underperformance of XLNet-large suggests a need to investigate how model size interacts with limited task-specific fine-tuning data.
  • Feature Interaction: While permutation importance was used, the authors note that it can overemphasize correlated features. Alternative methods like per-feature retraining were computationally intractable within the task timeframe.
  • Future Directions: The paper suggests further research into MWE representations and exploring how LLMs capture cognitive information.

Summary updated

BibTeX
@inproceedings{vickers2021cognlp,
  author = {Peter Vickers and Rosa Wainwright and Harish Tayyar Madabushi and Aline Villavicencio},
  title = {{CogNLP-Sheffield at CMCL 2021 Shared Task: Blending Cognitively Inspired Features with Transformer-based Language Models for Predicting Eye Tracking Patterns}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure