CMCL 2021 Shared Task on Eye-Tracking Prediction

Nora Hollenstein, Emmanuele Chersoni, Cassandra L. Jacobs, Yohei Oseki, Laurent Prévot, Enrico SantusNAACL 2021

Sure, here's a blog-style summary of the paper "CMCL 2021 Shared Task on Eye-Tracking Prediction":

TL;DR

This paper introduces the first shared task focused on predicting token-level eye-tracking metrics during natural reading. By standardizing evaluation on the ZuCo dataset, it provides a benchmark for computational models to better understand and replicate human reading processes, with the winning approach leveraging linguistic and psychometric features in a gradient boosting framework.

What Problem Does This Paper Address?

Understanding how humans process language is a fundamental goal in linguistics and natural language processing (NLP). Eye-tracking provides millisecond-accurate insights into where readers focus their attention, offering a direct window into cognitive processes. However, accurately modeling these gaze patterns computationally has been challenging. This paper addresses the need for a standardized evaluation framework to compare different approaches for predicting these eye-tracking metrics, aiming to advance our understanding of language comprehension and improve NLP models by incorporating human cognitive signals.

Methods

The core of this work is the establishment of a shared task. Participants were tasked with predicting five specific eye-tracking metrics for each token in a sentence:

  • NFIX (number of fixations): How many times a reader fixated on a word.
  • FFD (first fixation duration): The time spent on the first fixation of a word.
  • TRT (total reading time): The total time spent on a word, including regressions.
  • GPT (go-past time): The time spent on a word before moving to the next, including regressions to previous words.
  • FIXPROP (fixation proportion): The likelihood that a word was fixated by participants.

Submissions were evaluated using Mean Absolute Error (MAE). The paper notes that participating systems employed a range of methods, including:

  • Boosting algorithms with handcrafted features (e.g., CatBoost, LightGBM).
  • Neural network-based approaches, particularly fine-tuning transformer language models (e.g., BERT, RoBERTa, XLNet).
  • Hybrid systems combining both feature-based and neural approaches.

The winning system notably utilized a gradient boosting framework with a combination of linguistic and psychometric features.

Datasets

The primary dataset used for this shared task is the Zurich Cognitive Language Processing Corpus (ZuCo). This corpus is a valuable resource as it combines eye-tracking and EEG brain activity data from naturalistic reading experiments. It contains anonymized records and is compliant with ethical board approvals. The paper states that the ZuCo dataset used for the task comprises 990 sentences, split into 800 for training and 190 for testing. The sentences are drawn from movie reviews (Stanford Sentiment Treebank) and Wikipedia.

The paper also mentions other existing eye-tracking corpora for English, such as the Ghent Eye-Tracking Corpus (GECO) and the Provo corpus, as well as corpora for other languages, indicating a growing interest in this data modality.

Key Insights and Ideas

  • The power of linguistic and psychometric features: The winning system's success with a gradient boosting framework using these features suggests that well-crafted linguistic and psychometric information can be highly effective for predicting eye-tracking metrics, even without relying solely on large neural models.
  • Varying predictability across metrics: The paper highlights that FFD is the most accurately predicted feature, implying that models are better at capturing early lexical access stages than later, more complex semantic integration processes (reflected in TRT and NFIX).
  • Variance as a predictor of difficulty: Features with lower variance (like FFD, GPT, TRT) generally achieved lower MAEs, suggesting that less noisy data leads to more predictable outcomes. FIXPROP, with its higher variance, was more challenging but also saw the largest improvement from the systems compared to the baseline.
  • Standardization drives progress: The establishment of a shared task and a standardized benchmark (ZuCo) is crucial for comparing diverse approaches and fostering advancements in the field of cognitive modeling and computational linguistics.

Key Results

The paper presents the results of the shared task in Table 3. The winning team, LAST (Bestgen, 2021), achieved the lowest average Mean Absolute Error (MAE) across all five eye-tracking features, with an overall MAE of 3.813. This was significantly better than the mean baseline, which had an MAE of 7.357.

Specifically, the winning system achieved the following MAE scores:

  • NFIX: 3.879
  • FFD: 0.655
  • GPT: 2.197
  • TRT: 1.524
  • FIXPROP: 10.812

The paper notes that FFD was the most accurately predicted feature across most systems, while FIXPROP was the most challenging.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Limited test set size: The test set, while providing a benchmark, is relatively small (190 sentences), which might limit the generalizability of findings.
  • Need for more data: They suggest extending the dataset to include more English corpora and data from other languages (e.g., Dutch, Russian) to improve robustness and cross-lingual applicability.
  • Development set: Providing an official development set during the training phase would be beneficial for participants.
  • Additional evaluation metrics: Incorporating metrics like R² could offer a more nuanced understanding of model performance beyond MAE.
  • Hybrid approaches: The paper hints at the potential of hybrid systems that effectively combine linguistic features with state-of-the-art language models.

The shared task itself is seen as a significant step towards a standardized evaluation framework, paving the way for more rigorous testing and comparison of models aiming to replicate human reading behavior.

Summary updated

BibTeX
@inproceedings{hollenstein2021cmcl,
  author = {Nora Hollenstein and Emmanuele Chersoni and Cassandra L. Jacobs and Yohei Oseki and Laurent Prévot and Enrico Santus},
  title = {{CMCL 2021 Shared Task on Eye-Tracking Prediction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure