TL;DR
This paper tackles the challenge of predicting human eye-tracking patterns during reading across different languages, even for languages not seen during training (zero-shot cross-lingual prediction). They achieve state-of-the-art results by combining powerful multilingual transformer models with carefully engineered linguistic features, demonstrating the potential of these models for understanding cognitive processes in a language-agnostic way.
What Problem Does This Paper Address?
Understanding how humans read is crucial for cognitive science and natural language processing. Eye-tracking data, which records where a person looks while reading, offers a millisecond-level insight into cognitive processes like language comprehension. However, reading behavior can be highly language-specific due to differences in grammar and word order. This paper addresses the challenge of building models that can predict these eye-tracking patterns not just within a single language, but across multiple languages, and importantly, generalize to entirely new languages without explicit training data for them. This "zero-shot cross-lingual" capability is a significant hurdle, as it requires models to learn universal reading principles rather than language-specific ones.
Methods
The core of the approach involves a fusion model. First, they leverage pre-trained multilingual transformer models: mBERT, XLM, and XLM-ROBERTa. These models are trained on vast amounts of text from many languages and provide rich contextualized embeddings for words. To supplement these powerful embeddings, they engineer three additional linguistic features:
- tok_len: The number of sub-word tokens a word is broken into by the tokenizer, serving as a proxy for word complexity.
- word_char_len: The character length of a word in UTF-8 format, reflecting its visual space.
- rel_len: The relative length of a word compared to the preceding word, potentially capturing reading ease after a longer word.
These transformer embeddings and engineered features are then fed into a regression model to predict two key eye-tracking metrics: First Fixation Duration (FFD) and Total Reading Time (TRT), along with their standard deviations across readers. They experimented with different transformer models and combinations of their engineered features.
Datasets
The study utilizes data from the CMCL 2022 Shared Task, which comprises eye-tracking data from 8 datasets across 6 languages: Chinese, Dutch, English, German, Hindi, and Russian. The task is divided into two subtasks:
- Subtask 1: Predicting eye-tracking features for sentences from the 6 provided languages.
- Subtask 2: Predicting eye-tracking features for sentences from a new, unseen "surprise" language, testing the zero-shot cross-lingual generalization.
The data is scaled between 0 and 100, and feature values are averaged over readers.
Key Insights and Ideas
- Multilingual Transformers Excel: The paper strongly supports the efficacy of large multilingual transformer models (like XLM-ROBERTa) for cross-lingual language processing tasks, even in a zero-shot setting.
- Feature Fusion is Key: Simply using transformer embeddings wasn't enough. The addition of carefully selected linguistic features (tok_len, word_char_len, rel_len) significantly improved performance, particularly for predicting the mean values of eye-tracking metrics.
- Different Features for Different Metrics: The ablation study revealed that
word_char_lenhad a substantial impact on predictingFFDAvg, whilerel_lenwas more influential forSubTask-2performance, suggesting that different linguistic cues might be more relevant for different aspects of reading behavior. - Larger Models, Better Performance: Models with larger parameter counts (e.g., XLM-ROBERTa-large) generally outperformed their smaller counterparts, indicating that more capacity is beneficial for capturing complex cross-lingual patterns.
- XLM's Zero-Shot Advantage: While mBERT performed better on SubTask-1 (seen languages), XLM models showed superior performance on SubTask-2 (zero-shot), highlighting their strength in generalizing to unseen languages.
Key Results
The authors' best model, based on XLM-ROBERTa-large, achieved 4th place for SubTask-1 and 1st place for SubTask-2 in the CMCL 2022 Shared Task. Their transformer-based models consistently outperformed simpler baseline approaches that relied solely on token-level features. For instance, the best performing model achieved MAE scores of 4.738 for FFDAvg and 7.268 for TRTAvg on the dev set for SubTask-1, and 2.945 for FFDAvg and 5.602 for TRTAvg on the test set for SubTask-2.
Limitations and Future Work
The authors acknowledge that while their engineered features helped, they might not be exhaustive. They suggest future work could explore:
- More etymological features: Leveraging shared language history could provide deeper insights.
- Cross-language lexical similarity indices: Incorporating explicit measures of word similarity across languages could further enhance generalization.
- Surprisal Index: Integrating the concept of surprisal (how unexpected a word is) could positively correlate with reading time and fixation duration, potentially improving model performance.
Summary updated