A Machine Learning Approach for Detecting Cognitive Interference Based on Eye-Tracking Data

Antonio Rizzo, Sara Ermini, Dario Zanca, Dario Bernabini, Alessandro RossiFrontiers in Human Neuroscience 2022 · DOI

TL;DR

This paper demonstrates that machine learning models can accurately distinguish between cognitive tasks with and without interference using only eye-tracking data. By analyzing features of eye fixations and saccades, the researchers show that subtle patterns in visual behavior can reveal underlying cognitive load, paving the way for objective, non-invasive assessment of cognitive states.

What Problem Does This Paper Address?

The Stroop test is a classic psychological paradigm used to measure cognitive interference – the difficulty in processing information when conflicting stimuli are presented (e.g., the word "blue" printed in red ink). While the Stroop effect is well-established, objectively quantifying the cognitive load and interference associated with such tasks, especially in real-time, remains a challenge. Traditional methods rely on reaction times and error rates, which can be influenced by various factors. Eye movements, however, are known to be sensitive indicators of attention and cognitive load. This paper aims to bridge this gap by exploring whether eye-tracking data, specifically features related to fixations and saccades, can be used to build machine learning models capable of automatically detecting cognitive interference during different visual tasks.

Methods

The researchers designed a visual Stroop task with four conditions: "Reading Without Interference" (RWoI), "Reading With Interference" (RWI), "Naming Without Interference" (NWoI), and "Naming With Interference" (NWI). Eye movements were recorded from 64 participants using an EyeLink Portable Duo eye tracker at a 500 Hz sampling rate.

After data collection, a comprehensive set of eye-tracking features were extracted, categorized into:

  • Fixation Features: Number of fixations, average/maximum fixation duration, and horizontal/vertical regressions (indicating backward eye movements).
  • Saccade Features: Frequency of saccades in different directions, minimum/average/maximum saccade duration, velocity, amplitude, angle, distance, and slope.

These features were then used to train four different machine learning classifiers: Random Forests (RF), Logistic Regression, Artificial Neural Networks (ANN), and Support Vector Machines (SVM). The models were evaluated using a 5-fold cross-validation approach, focusing on binary classification tasks to distinguish between specific conditions (e.g., NWoI vs. RWoI, NWI vs. NWoI, RWI vs. RWoI). To account for significant inter-subject variability in eye movements, a subject-wise normalization technique was applied, where the mean of each variable was subtracted from the original values for each subject.

Datasets

The study utilized a custom dataset collected from 64 participants performing the designed visual Stroop tasks. This dataset, comprising eye-tracking data (fixations and saccades) for each of the four experimental conditions, serves as the benchmark for the machine learning models. The paper does not introduce a publicly available benchmark dataset but rather uses its collected data for evaluation.

Key Insights and Ideas

  • Eye Fixations as a Strong Indicator: The study strongly suggests that features related to eye fixations are particularly effective in distinguishing between different tasks and cognitive loads. Specifically, fixation duration and regressions (backward eye movements) showed significant differences, aligning with the hypothesis that increased cognitive load leads to more deliberate and potentially corrective eye movements.
  • Saccades Less Informative Alone: In contrast to fixations, saccade-related features alone did not consistently provide strong discriminative power between tasks or conditions. This implies that while saccades are crucial for visual exploration, their statistical properties might be less directly indicative of cognitive interference compared to the temporal and spatial characteristics of fixations.
  • Machine Learning for Objective Assessment: The success of machine learning models in classifying tasks based on eye-tracking data highlights the potential for objective, automated assessment of cognitive states. This moves beyond subjective reports or solely reaction-time measures.
  • Importance of Feature Combination: Combining features from both fixations and saccades generally led to improved classification performance across all tested models, suggesting that a holistic view of visual behavior is more informative than isolated feature sets.
  • Subject-Wise Normalization is Crucial: The application of subject-wise normalization significantly improved model performance, underscoring the substantial inter-subject variability in eye-tracking data and the need for methods that can account for individual differences.

Key Results

The machine learning models achieved good classification performances, consistently outperforming the random baseline (0.5 accuracy).

  • Accuracy: For instance, in distinguishing between Naming Without Interference (NWoI) and Naming With Interference (NWI), models using combined fixation and saccade features (Fix+Sac) achieved accuracies well above 0.7, with some reaching close to 0.8. Similar high accuracies were observed for other classification pairs.
  • F1-Scores: The F1-scores, which balance precision and recall, also demonstrated strong performance, with many models achieving scores above 0.7 and some exceeding 0.8, particularly when using combined features.
  • ANOVA Results: Traditional statistical analysis (ANOVA) on fixation features also revealed significant differences between tasks and conditions, supporting the machine learning findings. For example, variables like fixation duration and regressions showed significant p-values when comparing Naming vs. Reading tasks.

Limitations and Future Work

  • Sample Size: The authors acknowledge that the dataset size (64 subjects) is limited for advanced machine learning studies, particularly for deep learning approaches. This limitation might contribute to the observed variability in results.
  • Generalizability: While the models performed well on this specific dataset, further validation on larger and more diverse datasets would be necessary to ensure generalizability.
  • Feature Engineering: The study relied on hand-crafted features. Future work could explore end-to-end deep learning approaches that automatically learn relevant features from raw eye-tracking data.
  • Comparison with Other Tasks: The authors suggest comparing gaze behavior in the Stroop task with other tasks that induce cognitive conflict (e.g., Simon task) to further understand the mechanisms of interference.
  • Integration of Pupillary Response: Incorporating pupillary response data, which is known to correlate with attention and cognitive load, could provide a more comprehensive understanding of cognitive states.
  • Practical Applications: The findings suggest potential applications in areas like monitoring driver attention, detecting deception, or developing more adaptive human-computer interfaces.

Summary updated

Abstract

The Stroop test evaluates the ability to inhibit cognitive interference. This interference occurs when the processing of one stimulus characteristic affects the simultaneous processing of another attribute of the same stimulus. Eye movements are an indicator of the individual attention load required for inhibiting cognitive interference. We used an eye tracker to collect eye movements data from more than 60 subjects each performing four different but similar tasks (some with cognitive interference and some without). After the extraction of features related to fixations, saccades and gaze trajectory, we trained different Machine Learning models to recognize tasks performed in the different conditions (i.e., with interference, without interference). The models achieved good classification performances when distinguishing between similar tasks performed with or without cognitive interference. This suggests the presence of characterizing patterns common among subjects, which can be captured by machine learning algorithms despite the individual variability of visual behavior.
BibTeX
@inproceedings{rizzo2022machine,
  author = {Antonio Rizzo and Sara Ermini and Dario Zanca and Dario Bernabini and Alessandro Rossi},
  title = {{A Machine Learning Approach for Detecting Cognitive Interference Based on Eye-Tracking Data}},
  booktitle = {Frontiers in Human Neuroscience},
  year = {2022},
  doi = {10.3389/fnhum.2022.806330},
}
Figure