TL;DR
This paper presents a successful approach to predicting human eye-tracking data during reading across six diverse languages. By leveraging a gradient boosting framework with carefully selected linguistic features, including word surprisal, the authors achieved the best performance in the CMCL 2022 Shared Task, demonstrating the power of a unified, feature-rich model for multilingual reading prediction.
What Problem Does This Paper Address?
Understanding how humans process language during reading is a fundamental goal in psycholinguistics. Eye-tracking data, which records where and for how long a person looks at words, provides a rich source of information about these cognitive processes. However, predicting this data accurately is challenging, especially across multiple languages with different linguistic structures and writing systems. The CMCL 2022 Shared Task aimed to address this by asking participants to predict four key eye-tracking features (First Fixation Duration, its standard deviation, Total Reading Time, and its standard deviation) for sentences in six languages. This paper tackles the challenge of building a single, robust model that can generalize across these diverse linguistic landscapes.
Methods
The core of the proposed method is a gradient boosting regressor (specifically, LightGBM, or LGB). This powerful machine learning algorithm is well-suited for complex prediction tasks. The authors engineered a comprehensive set of linguistic features to feed into this model. These features fall into several categories:
- Word-level features: This includes basic properties of the target word and its preceding word, such as word position, word length, and whether the word is capitalized or entirely in uppercase.
- Frequency-based features: Word frequencies for both the current and previous words were incorporated, leveraging the
wordfreqlibrary. - Syllable counts: This feature was included for Hindi using a specialized library, and for other languages via
textstat. - Surprisal: A crucial feature is the "surprisal" of the target word, calculated as the negative logarithm of its probability given the preceding context. This was computed using various pre-trained GPT language models for each language.
To further enhance the model's ability to capture interactions between these features, the authors generated polynomial features of order 2. This means that combinations of two features were also fed into the regressor, potentially revealing more nuanced relationships.
Datasets
The study utilized materials from eight openly available eye movement corpora across six languages:
- Chinese: Beijing Sentence Corpus
- Dutch: GECO Corpus
- English: Provo Corpus, ZuCo 1.0 Corpus, ZuCo 2.0 Corpus
- German: Potsdam Textbook Corpus
- Hindi: Postdam-Allahabad Hindi Eyetracking Corpus
- Russian: Russian Sentence Corpus
The paper provides a table (Table 1) detailing the number of instances in the train, development, and test sets for each of these corpora.
Key Insights and Ideas
- Unified Model Power: The success of a single gradient boosting model across six diverse languages highlights the effectiveness of carefully chosen linguistic features and a robust learning algorithm in capturing cross-lingual reading patterns.
- Surprisal is Key: The inclusion of word surprisal, derived from language models, proved to be a highly effective feature for predicting eye-tracking metrics, underscoring the role of predictability in reading.
- Feature Interactions Matter: Generating polynomial features to capture interactions between basic linguistic features and surprisal significantly improved performance, suggesting that these features are not independent predictors.
- FFD vs. TRT: The study observed a tendency for better prediction of First Fixation Duration (FFD) related features compared to Total Reading Time (TRT). This might be because FFD is more directly linked to lexical processing, while TRT can be influenced by later-stage cognitive processes.
- Language-Specific Nuances: While the unified model performed well, the authors acknowledge that certain features or models might be more suitable for specific languages, suggesting avenues for future language-specific tuning.
Key Results
The authors' gradient boosting system achieved the lowest Mean Absolute Error (MAE) among all participating systems in the CMCL 2022 Shared Task Subtask 1, making it the best system of the competition. Specifically, the paper reports an average MAE of 3.0112 across all predictions. Table 2 shows that for Hindi, the LGB regressor achieved the lowest MAE for FFD, outperforming other regressors. Table 3 provides a detailed breakdown of the best-performing models for each language and feature, with LGB variants frequently appearing. The ablation studies (Tables 4 and 5) indicate that most of the incorporated features, including capitalization, contribute positively to the prediction accuracy.
Limitations and Future Work
The authors acknowledge that predicting TRTAvg was the most challenging feature. They also note that while their unified model performed well, there's room for improvement by identifying regression features and models that are more specifically suited to individual languages. Future work could involve a more comprehensive cross-lingual comparison of features and models. Additionally, exploring more advanced language models for surprisal calculation or investigating deeper interaction features could further enhance prediction accuracy.
Summary updated