Native Language Prediction from Gaze: a Reproducibility Study

Lina Skerath, Paulina Toborek, Anita Zielińska, Maria Barrett, Rob van der GootACL 2023

TL;DR

This paper successfully replicates a previous study showing that a person's native language can be predicted from their eye-tracking data while reading English. While achieving comparable accuracy with a more diverse set of languages, the study found that the relationship between linguistic similarity and reading patterns might be more complex than initially thought.

What Problem Does This Paper Address?

It's well-established that a person's native language influences how they learn and process foreign languages. Eye movements, captured through eye-tracking, offer a window into these cognitive processes in real-time. While previous research has hinted at differences in eye-tracking patterns based on native language, a key question is whether these subtle differences are robust enough to predict a person's native language using machine learning. A prior study by Berzak et al. (2017) demonstrated this possibility using eye-tracking data from English readers with four different native language backgrounds. However, to truly validate this finding and explore its generalizability, a replication with a larger and more linguistically diverse set of languages was needed. This paper addresses that gap by attempting to reproduce Berzak et al.'s results using a new, larger dataset.

Methods

The researchers aimed to replicate the methodology of Berzak et al. (2017). They employed a log-linear model for Native Language Identification from Reading (NLIR). The core of their approach involved extracting features from eye-tracking data, specifically focusing on fixation durations. They utilized three main feature sets:

  • Words in Fixed Context (WFC): This set directly uses fixation times for individual words without aggregation.
  • Syntactic Clusters (SC): This set aggregates fixation times based on predicted syntactic information (like UPOS labels, POS tags, and dependency labels).
  • Information Clusters (IC): This set groups features based on word length, summing fixation times for words of similar lengths.

They also explored a combination of SC and IC features. To assess the relationship between linguistic similarity and reading patterns, they calculated two similarity metrics:

  • Linguistic-based similarity: Derived from typological data about languages.
  • Model-based similarity (English Reading Similarity - ERS): Based on the uncertainty of the NLIR classifier when predicting one language given another.

Datasets

The primary dataset used in this study is the Multilingual Eye-movement Corpus (MECO) L2 dataset (Kuperman et al., 2022). This dataset contains eye-tracking input from native speakers of 12 languages reading an English encyclopedic text, collected across 12 different laboratories. For their study, they selected a subset of seven languages with the most participants. To avoid overfitting, they randomly undersampled participants from the two largest cohorts to match the size of the third largest group.

The linguistic similarity data was obtained from the URIEL Typological Compendium (Littell et al., 2017a) and supplemented with data from Glottolog's world language tree.

Key Insights and Ideas

  • Reproducibility Confirmed: The study successfully replicates the core finding of Berzak et al. (2017), demonstrating that native language can indeed be predicted from eye-tracking data during L2 reading. This strengthens the evidence for cross-linguistic influence on reading processes.
  • Broader Language Coverage: By using a dataset with seven more interrelated languages compared to the original study's four, the findings are more robust and generalizable.
  • Data Efficiency: Surprisingly, the researchers achieved comparable classification performance even when using less data per participant (595 words vs. 900 words in the original study).
  • Complexity in Similarity: A key finding is that the relationship between linguistic similarity and reading patterns, as measured by classification uncertainty, appears more complex than initially suggested by Berzak et al. (2017). The expected positive correlation was not clearly observed in this study.
  • Feature Set Performance: While the "Words in Fixed Context" (WFC) feature set performed best, the trend of improvements with additional bigram and trigram features seen in the original paper was not replicated here.

Key Results

  • The log-linear model achieved a maximum cross-validation accuracy of 81.29% using the WFC feature set, which is comparable to the original study's best performance.
  • Despite using a more diverse set of seven languages, the model performed substantially better than a majority class baseline across all feature sets.
  • The study found no clear trend or significant correlation between linguistic similarity (from URIEL) and the model-based classification uncertainty (ERS), contrary to the original study's findings. Hierarchical clustering also showed little overlap between groupings based on linguistic similarity and classification uncertainty.

Limitations and Future Work

  • Dataset Limitations: The MECO dataset, while collected under a strict protocol, was recorded in different labs. This introduces potential confounding factors like location and experimenter effects, although the authors note that the original study's dataset also had limitations. The lack of other comparable datasets makes full replication challenging.
  • Nuanced Relationship: The primary limitation is the failure to replicate the strong correlation between linguistic similarity and reading patterns. This suggests that the relationship is likely more nuanced and might depend on specific linguistic features or the chosen similarity metrics.
  • Future Work: The authors suggest further investigation into the complex relationship between linguistic similarity and reading patterns. Exploring other feature sets, more sophisticated models, and potentially different similarity metrics could provide deeper insights. Additionally, investigating the impact of location and experimenter effects within the MECO dataset could refine future studies.

Summary updated

BibTeX
@inproceedings{skerath2023native,
  author = {Lina Skerath and Paulina Toborek and Anita Zielińska and Maria Barrett and Rob van der Goot},
  title = {{Native Language Prediction from Gaze: a Reproducibility Study}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure