MTL782_IITD at CMCL 2021 Shared Task: Prediction of Eye-Tracking Features Using BERT Embeddings and Linguistic Features

Shivani Choudhary, Kushagri Tandon, Raksha Agarwal, Niladri ChatterjeeNAACL 2021

TL;DR

This paper tackles the challenge of predicting human eye-tracking features (like fixation duration and count) from text using machine learning. By combining powerful pre-trained language models (BERT) with linguistic features, the researchers developed models that achieved competitive performance on a shared task, offering insights into how we process language.

What Problem Does This Paper Address?

Understanding how humans read and comprehend text is a fundamental goal in cognitive science and natural language processing. Eye movements during reading serve as a direct window into this process, revealing which parts of a sentence capture our attention and how long we dwell on them. Predicting these eye-tracking metrics for any given word in a sentence is a complex task because it requires capturing subtle linguistic nuances and cognitive processes. This paper addresses the challenge of building accurate predictive models for five key eye-tracking features: nFix (number of fixations), FFD (first fixation duration), TRT (total reading time), GPT (gaze past time), and fixProp (fixation proportion).

Methods

The researchers experimented with three main types of machine learning models:

  • Gradient Boosting (LGBM): They utilized LightGBM, a highly efficient gradient boosting framework, known for its speed and performance on large datasets.
  • Artificial Neural Networks (ANN): A deep ANN with seven layers was implemented, employing a standard architecture with ReLU activations and Adam optimization.
  • Convolutional Neural Networks (CNN): A CNN model was developed to capture contextual information by using a sliding window approach to process surrounding words. This involved separate branches for left and right context, followed by convolutional and pooling layers.

All models were trained using a combination of:

  • BERT Embeddings: Pre-trained contextual word embeddings from BERT were used to capture semantic meaning.
  • Linguistic Features: These included word length, word ID (position in the sentence), and syllable count. Additional features like POS tags and dependency labels were explored but ultimately excluded due to lack of performance improvement.

Datasets

The primary dataset used for this research was the Zurich Cognitive Language Processing Corpus (ZuCo 1.0 and ZuCo 2.0). This corpus provides simultaneous EEG and eye-tracking data during natural sentence reading. The dataset was divided into training and testing sets. For validation purposes, the first 100 sentences from the training data were utilized.

Key Insights and Ideas

  • BERT is a Powerful Foundation: The inclusion of BERT embeddings was crucial for achieving good performance across all models, highlighting the importance of pre-trained language understanding for predicting cognitive processes.
  • Linguistic Features Matter: Simple linguistic features like word length and position (Word_id) played a significant role, suggesting that structural aspects of words and sentences are important cues for eye-tracking.
  • Model Complexity vs. Performance: While CNNs are architecturally more complex, the LGBM regressor often achieved comparable or even better results, indicating that simpler, well-tuned models can be highly effective for this task.
  • Feature Importance Varies: The impact of individual features differed across different eye-tracking metrics. For instance, POS tags had a minimal impact on overall MAE for CNN models, and adding syllable counts didn't always improve LGBM performance. This suggests that feature selection might need to be tailored to specific metrics.
  • Context Window Size is Key for CNNs: For the CNN model, a context window size of K=10 proved to be optimal for capturing relevant contextual information.

Key Results

The CNN model using the feature set "Word_id + Length + BERT" achieved the best overall performance, with an Average Mean Absolute Error (Avg_MAE) of 3.99. This was a competitive result in the CMCL 2021 shared task, ranking 7th. The average MAE was further reduced to 3.994 in post-task evaluation. The LGBM models also performed well, particularly in predicting nFix, FFD, and fixProp.

Limitations and Future Work

  • Hyperparameter Tuning: The authors acknowledge that rigorous hyperparameter tuning was not fully explored, especially for the CNN and ANN models. They suggest that further tuning could potentially improve performance.
  • Feature Space Customization: The paper indicates that curating the feature space separately for each individual eye-tracking feature could lead to better accuracy.
  • Model Complexity: While CNNs are complex, their performance was not consistently superior to simpler models like LGBM, suggesting that further research into optimizing CNN architectures for this specific task might be beneficial.
  • Rigorous Tuning: The authors explicitly state that they did not perform rigorous hyperparameter tuning, which they suggest could be a direction for future work.

Summary updated

BibTeX
@inproceedings{choudhary2021mtl,
  author = {Shivani Choudhary and Kushagri Tandon and Raksha Agarwal and Niladri Chatterjee},
  title = {{MTL782\_IITD at CMCL 2021 Shared Task: Prediction of Eye-Tracking Features Using BERT Embeddings and Linguistic Features}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure