Team ÚFAL at CMCL 2022 Shared Task: Figuring out the correct recipe for predicting Eye-Tracking features using Pretrained Language Models

Sunit Bhattacharya, Rishu Kumar, Ondrej BojarACL 2022

TL;DR

This paper explores how to best leverage pre-trained language models (PLMs) like BERT and XLM for predicting human eye-tracking data during reading. The researchers found that incorporating contextual information, augmenting with linguistic features, and fine-tuning the models significantly improve prediction accuracy, achieving a top-5 ranking in the CMCL 2022 Shared Task.

What Problem Does This Paper Address?

Understanding human cognition, particularly how we comprehend language, is a long-standing research goal. Eye-tracking data, which captures where and for how long a person looks at words while reading, offers a rich source of information about this cognitive process. However, effectively translating this complex, high-dimensional data into actionable insights using computational models has been challenging. This paper tackles the problem of predicting specific eye-tracking features (like Total Reading Time and First Fixation Duration) from reading data, specifically within a multilingual context, by investigating the optimal use of modern pre-trained language models.

Methods

The authors experimented with various configurations of pre-trained multilingual language models, primarily BERT and XLM. Their approach involved:

  • Contextual Information: They compared models that received no additional context (System-1) with those that received the preceding words in a sentence as context (System-2).
  • Pooling Strategies: Different methods were used to aggregate token-level representations from the PLMs into a sentence-level representation. This included using the final hidden state of the first sub-word token, mean-pooling, and sum-pooling across all sub-word tokens.
  • Fine-tuning vs. Regression Layer: They explored two main training strategies: freezing the PLM weights and adding a regression layer (classifier), versus fine-tuning the entire model with a regression layer on top.
  • Augmenting Features: Beyond the standard linguistic embeddings from PLMs, they augmented the input with additional lexical features like word length and word frequency.

Datasets

The study utilized data from the CMCL 2022 Shared Task on Multilingual and Cross-lingual prediction of human reading behavior. This dataset comprised eye-tracking data for sentences in six languages (Chinese, Dutch, English, German, Hindi, Russian). The training set contained 1703 sentences, with development and test sets of 104 and 324 sentences, respectively. The eye-tracking features (mean and standard deviation of Total Reading Time and First Fixation Duration) were scaled between 0 and 100.

Key Insights and Ideas

  • Context is King (Mostly): Providing preceding sentence context (System-2) generally improved performance, though the impact wasn't always dramatic, and System-1 models still performed well. This suggests that while context is beneficial, its independent contribution might be less than expected when combined with powerful PLMs.
  • XLM Edges Out BERT: For this specific task, XLM-based models showed a slight advantage over BERT, achieving marginally lower MAE scores across several attributes.
  • Augmentation Pays Off: Adding linguistic features like word length and frequency to the PLM embeddings significantly improved prediction accuracy, reducing variance and MAE. This highlights the value of combining rich contextual representations with explicit linguistic knowledge.
  • Fine-tuning is Crucial: Fine-tuning the entire pre-trained model (rather than just training a regression layer on top) led to substantially better performance and reduced variance, reinforcing the standard practice in NLP.
  • Pooling Matters: The choice of pooling strategy for token representations had a noticeable impact, with mean-pooling and using the first sub-word token generally outperforming sum-pooling.

Key Results

The team's submissions achieved an average Mean Absolute Error (MAE) of 5.72, ranking them 5th in the CMCL 2022 Shared Task. Further post-task evaluation showed an improvement to an average MAE of 5.25. The best performing model, bert_sys2_augmented_sum_classifier, achieved an MAE of 5.251. The paper also details attribute-wise performance, noting that "TRT_Avg" (average Total Reading Time) exhibited the highest MAE and variance across models.

Limitations and Future Work

The authors acknowledge that while System-2 models (with context) generally performed better, the difference wasn't always substantial, suggesting that the independent boost from context alone might be limited. They also note that their experiments were guided by intuition and literature surveys, implying that a more exhaustive search of hyperparameter space or model configurations could potentially yield further improvements. Future work could explore more sophisticated ways of integrating linguistic features, investigate different pooling mechanisms, and potentially extend these methods to other cognitive tasks or languages.

Summary updated

BibTeX
@inproceedings{bhattacharya2022team,
  author = {Sunit Bhattacharya and Rishu Kumar and Ondrej Bojar},
  title = {{Team ÚFAL at CMCL 2022 Shared Task: Figuring out the correct recipe for predicting Eye-Tracking features using Pretrained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure