Mind Wandering in a Multimodal Reading Setting: Behavior Analysis & Automatic Detection Using Eye-Tracking and an EDA Sensor

Iuliia Brishtel, Anam Ahmad Khan, Thomas Schmidt, Tilman Dingler, Shoya Ishimaru, Andreas Dengel2020 · DOI

TL;DR

This paper explores how text content and music influence mind-wandering during reading and develops a multimodal approach using eye-tracking and electrodermal activity (EDA) to automatically detect these attention drifts. The findings suggest that combining these sensors significantly improves detection accuracy, paving the way for more adaptive learning systems.

What Problem Does This Paper Address?

Mind-wandering, the shift of attention away from a task to internal thoughts, is a common phenomenon that can hinder learning and productivity. While task-related thoughts can foster creativity, task-unrelated thoughts are detrimental to comprehension. Detecting mind-wandering is challenging due to its covert nature. Existing methods like experience sampling can be disruptive. While physiological sensors like EEG and fMRI have been used, they are expensive. Simpler methods like eye-tracking and EDA have shown promise, but there's a need for more robust, user-independent detection systems, especially for educational settings. Furthermore, the interplay of external factors like text semantics and music on mind-wandering, and how these can be leveraged for detection, remains an area needing further investigation.

Methods

The study employed a multimodal approach combining eye-tracking and electrodermal activity (EDA) for mind-wandering detection.

  • Behavioral Analysis: The researchers investigated the effects of two independent variables: Text Type (Computer Science, Psychology, Random Topic) and Music Type (Sad, Happy, No-Music) on mind-wandering frequency. They used a repeated-measures design where participants experienced all conditions. Behavioral data, including self-reports on interest, difficulty, tiredness, attentional focus, and type of thoughts (task-related vs. task-unrelated), were collected.
  • Physiological Data Acquisition:
    • Eye-Tracking: A Tobii 4C gaming eye-tracker was used to record gaze points at 90 Hz. Features extracted included fixation duration, pupil size, saccade length, velocity, and angle, as well as regression metrics.
  • EDA: An Empatica E4 wristband sampled EDA at 4 Hz. The EDA signal was decomposed using a convex optimization approach into tonic and phasic components. Statistical features were then extracted from the raw, tonic, and phasic components.
  • Machine Learning Models: Three classifiers were used for automatic mind-wandering detection: Logistic Regression (baseline), Support Vector Machine (SVM), and Random Forest. Models were trained using a leave-one-participant-out cross-validation strategy to ensure user-independence. To address the class imbalance (mind-wandering occurred in only 15.8% of paragraphs), the Synthetic Minority Over-sampling Technique (SMOTE) was applied to the training data. Feature importance was analyzed using the SHAP method.

Datasets

The study utilized a dataset collected from 21 graduate and undergraduate students. Participants read 12 scientific texts in German, split into paragraphs, under various music conditions. The texts were categorized into Computer Science, Psychology, and Random Topics. The collected data included:

  • Self-reported data on interest, difficulty, tiredness, attentional focus, and type of thoughts.
  • Eye-tracking data (gaze points, pupil diameter).
  • Electrodermal activity (EDA) data.

No new benchmark datasets were introduced; the study used existing texts and collected data from a controlled experiment.

Key Insights and Ideas

  • Music's Disruptive Power: Contrary to some expectations, both sad and happy music increased mind-wandering frequency compared to no music. This suggests that any auditory stimulus, especially unfamiliar music, can be a significant distractor during reading.
  • Text Relevance Nuance: While the hypothesis that personally or academically relevant texts would increase mind-wandering wasn't fully supported, an interaction effect was observed: sad music combined with academically relevant texts (Computer Science) led to more mind-wandering. This highlights the complex interplay between external stimuli and internal relevance.
  • EDA as a Powerful Indicator: The study demonstrated that EDA, particularly when decomposed into its tonic and phasic components, holds significant potential for mind-wandering detection, even as a single sensor.
  • Synergy of Sensors: Combining eye-tracking and EDA data yielded the highest detection accuracy, suggesting that these modalities capture complementary aspects of cognitive state (visual attention vs. emotional arousal).
  • Behavioral Data's Limited Role: While reading duration showed some utility, other behavioral self-reports did not significantly improve the classification accuracy when combined with sensor data, indicating that physiological signals are sufficient for robust detection.

Key Results

  • Mind Wandering Frequency: Music (both sad and happy) significantly increased mind-wandering frequency compared to the no-music condition. An interaction effect showed sad music with academically relevant texts (Computer Science) led to the highest mind-wandering.
  • Task-Related Thoughts (TRTs): Participants reported more TRTs when reading random topics, possibly indicating a need to engage with the material due to low prior knowledge.
  • Detection Performance:
    • An EDA-only Random Forest model achieved an F₁-Score of 0.78.
  • An eye-tracking-only Random Forest model achieved an F₁-Score of 0.80.
  • The combined Eye + EDA Random Forest model achieved the highest F₁-Score of 0.83.
  • The combination of Eye + EDA features improved classification accuracy by 3% over the eye-tracking-only model.

Limitations and Future Work

  • Self-Caught Method: The reliance on the self-caught method for ground truth means detection accuracy is limited by participants' ability to accurately monitor and report their thoughts.
  • Offline Detection: The current method is an offline detection technique. Due to the low sampling rate of EDA sensors and their response delay, real-time, moment-to-moment detection is challenging.
  • Binary Classification: The study focused on detecting the presence or absence of mind-wandering. Future work should investigate distinguishing between task-related and task-unrelated thoughts, which is crucial for pedagogical applications.
  • Spectral Analysis for EDA: The study did not employ spectral analysis for EDA features, which could potentially offer further insights and improve classification accuracy.
  • Real-World Validation: The results were obtained in a controlled experimental setting. Future work should verify these findings in a real classroom environment.

Summary updated

BibTeX
@misc{brishtel2020mind,
  author = {Iuliia Brishtel and Anam Ahmad Khan and Thomas Schmidt and Tilman Dingler and Shoya Ishimaru and Andreas Dengel},
  title = {{Mind Wandering in a Multimodal Reading Setting: Behavior Analysis \& Automatic Detection Using Eye-Tracking and an EDA Sensor}},
  year = {2020},
  doi = {10.3390/s20092546},
}
Figure