TL;DR
This paper presents the CMCL 2022 Shared Task, which challenged researchers to predict human eye-tracking behavior across multiple languages, including a surprise language. The task introduced a novel focus on predicting not just average reading patterns but also individual reader variability, pushing the boundaries of cognitive modeling and natural language processing.
What Problem Does This Paper Address?
Understanding how humans read is fundamental to cognitive science and has implications for developing more human-like AI. Eye-tracking data provides rich, millisecond-accurate insights into this process, revealing where readers look and how long they fixate on words. However, predicting these patterns computationally, especially across different languages, is a significant challenge. Previous work often focused on single languages and average reader behavior. This paper addresses the need for a standardized evaluation framework and a high-quality dataset to advance multilingual and cross-lingual prediction of human reading behavior, accounting for both general trends and individual differences.
Methods
The CMCL 2022 Shared Task was structured as a regression problem. Participants were tasked with predicting four key eye-tracking features for each word in a sentence:
- FFDAVG (First Fixation Duration Average): The average duration of the first time a reader looks at a word.
- FFDSTD (First Fixation Duration Standard Deviation): The variability in the first fixation duration across different readers.
- TRTAVG (Total Reading Time Average): The total time a reader spends on a word, including re-fixations.
- TRTSTD (Total Reading Time Standard Deviation): The variability in total reading time across readers.
The task involved two subtasks:
- Subtask 1 (Multilingual Prediction): Predicting eye-tracking features for sentences in six provided languages using training data from those same languages.
- Subtask 2 (Cross-lingual Prediction): Predicting eye-tracking features for sentences in a new, unseen language (Danish) using the training data from Subtask 1.
Teams could use any freely available data sources for training, encouraging diverse approaches ranging from traditional regression models with linguistic features to advanced neural models like multilingual transformers.
Datasets
The shared task utilized several publicly available eye-tracking corpora for training and development, covering six languages: Chinese (BSC), Hindi (PAHEC), Russian (RSC), English (Provo, ZuCo 1.0, ZuCo 2.0), Dutch (GECO-NL), and German (PoTeC). For Subtask 2, a new Danish test set was introduced, sourced from the CopCo corpus. This multilingual and cross-lingual setup aimed to test the robustness and transferability of prediction models.
Key Insights and Ideas
- Individual Differences Matter: The inclusion of standard deviation for eye-tracking features (FFDSTD, TRTSTD) was a key innovation, forcing models to account for variability between readers, not just average behavior. This is crucial for building more cognitively plausible models.
- Multilingual Models Show Promise: The task highlighted the effectiveness of multilingual transformer embeddings for cross-lingual transfer, even for unseen languages, suggesting these models capture underlying linguistic universals.
- Lexical and Statistical Features Remain Relevant: While neural models performed well, simpler regression models leveraging lexical features (like word length, frequency, and surprisal) also achieved competitive results, indicating that well-crafted traditional features are still powerful predictors.
- The Challenge of Unseen Languages: Subtask 2 demonstrated the difficulty of predicting behavior in a completely new language, pushing teams to explore zero-shot learning and translation-based approaches.
Key Results
- Subtask 1 Winner (HkAmsters): Achieved the best performance using language-specific regression models based on word-level features, including word length, frequency, and surprisal estimated with GPT-2.
- Subtask 2 Winner (Poirot): Employed a zero-shot hybrid model combining multilingual transformer embeddings with statistical features.
- Baselines: A simple mean baseline (predicting the average value for each feature) served as a reference, with MAE scores ranging from 3.57 to 18.87 for Subtask 1 features.
Limitations and Future Work
- Dataset Size and Diversity: The authors acknowledge that the training and test sets, while diverse in text domains, were limited in size. This could impact the generalizability of models.
- Tokenization Nuances: The paper notes that tokenization followed the experimental presentation, not necessarily linguistic correctness, which might affect model performance.
- Future Directions: The authors express hope that the CMCL Shared Task will foster further research by providing a standard evaluation framework and a high-quality dataset for multilingual and cross-lingual reading prediction. Future work could involve exploring more diverse languages, larger datasets, and more sophisticated cognitive modeling approaches.
Summary updated