Accessible Real-time Eye-gaze Tracking for Neurocognitive Health Assessment: A Multimodal Web-based Approach

Daniel Tisdale, Jackson Liscombe, David Paulter, Michael Neumann, Vikram RamanarayananInterspeech 2025

TL;DR

This paper introduces a novel, accessible web-based system that uses standard webcams for real-time eye-gaze tracking to help assess neurocognitive health, specifically Mild Cognitive Impairment (MCI). By integrating this with a multimodal dialogue system, they achieved high accuracy in differentiating MCI patients from healthy controls, demonstrating a scalable and engaging approach to remote health monitoring.

What Problem Does This Paper Address?

Diagnosing and monitoring neurocognitive disorders like Mild Cognitive Impairment (MCI) traditionally relies on expensive, specialized equipment and controlled clinical settings. This makes regular, accessible health tracking difficult and limits early detection and intervention. The authors aim to bridge this gap by leveraging the ubiquity of webcams and advancements in real-time machine learning to create an affordable, scalable, and user-friendly system for remote health assessments.

Methods

The core of the system is the integration of WebGazer, a real-time eye-gaze tracking library that runs in the browser using TensorFlow.js and FaceMesh, into a multimodal dialogue platform. Unlike some previous WebGazer implementations that relied on cursor position, this system blocks cursor data during interactive tasks to improve accuracy and accessibility.

Key methodological innovations include:

  • Multimodal Fusion: Combining eye-gaze data with speech and other conversational cues from a virtual human guide (Tina) for a richer assessment.
  • Webcam-based Eye-Gaze: Utilizing standard webcams, eliminating the need for specialized hardware.
  • Browser-based Processing: Performing eye-gaze estimation directly in the user's browser for privacy and reduced server load.
  • Novel Eye-Gaze Tasks: Adapting clinically relevant ocular-motor exercises into interactive web tasks (e.g., Smooth Pursuit, Saccade tasks, Stroop task, Image Exploration) to elicit discriminative eye-gaze patterns.
  • Data Preprocessing and Normalization: Implementing a robust pipeline to handle variations in user window sizes and potential resizing, including fitting, normalization to [0,1], and Savitzky-Golay filtering for smoothing.
  • Feature Engineering: Deriving interpretable metrics from eye-gaze data, such as velocity, acceleration, distance from target, time to target, and number of times on target.
  • Feature Selection and Classification: Employing Kruskal-Wallis tests for feature ranking and using an AdaBoost classifier for binary classification between MCI and healthy controls.

Datasets

The study utilized a crowdsourced dataset comprising 39 participants: 10 with Mild Cognitive Impairment (MCI) and 29 healthy controls. Participants engaged in an approximately 10-minute interview with 11 interactive eye-gaze tasks. The data collected included raw video, facial landmarks, audio, survey demographics, and eye-gaze data. No new benchmark datasets were introduced; rather, existing data was collected and analyzed.

Key Insights and Ideas

  • Democratizing Eye-Gaze Tracking: The most significant idea is making sophisticated eye-gaze tracking accessible to anyone with a webcam, removing a major barrier to its use in healthcare.
  • Interpretability is Key: The authors prioritize extracting human-interpretable metrics from eye-gaze data rather than relying solely on "black box" deep learning models, which is crucial for clinical adoption.
  • Engagement Matters: High user engagement and positive user experience are critical for the practical adoption of such systems, and their platform achieved over 80% positive ratings.
  • Subtle Gaze Differences: Even with similar overall prediction errors, MCI participants showed distinct trends in their gaze patterns (e.g., being "off phase" in smooth pursuit tasks), highlighting the potential of these subtle differences for diagnosis.

Key Results

  • Classification Accuracy: An AdaBoost classifier, using features derived from the eye-gaze tasks, achieved an average accuracy of 0.94 in differentiating MCI participants from healthy controls.
  • User Experience: Over 80% of participants reported high engagement and liked the user experience with the platform.
  • Prediction Error: For healthy controls, the estimated eye-gaze prediction error was approximately 14.7% on the x-axis and 18.3% on the y-axis. For MCI participants, these errors were slightly higher at 15.3% and 19.3%, respectively.
  • Feature Importance: Metrics related to max and standard deviations of distances, velocities, and accelerations were found to be highly indicative of differences between cohorts.

Limitations and Future Work

Limitations:

  • Small Sample Size: The study had a relatively small sample size for the MCI cohort, which limits the ability to train complex deep learning models.
  • Limited Deep Learning Exploration: The authors did not extensively explore deep learning architectures for classification due to the sample size and interpretability concerns.

Future Work:

  • Larger and Diverse Datasets: Applying the system to larger sample sizes and a more diverse range of neurological, neurocognitive, and neurodevelopmental disorders.
  • Deep Learning Integration: Investigating how to leverage deep learning models for improved classification accuracy while maintaining or enhancing interpretability.
  • Longitudinal Studies: Utilizing the system for consistent monitoring in longitudinal studies to gain deeper insights into disease etiology and progression.
  • Further Model Explainability: Researching methods to make deep learning models more explainable for healthcare applications.

Summary updated

BibTeX
@inproceedings{tisdale2025accessible,
  author = {Daniel Tisdale and Jackson Liscombe and David Paulter and Michael Neumann and Vikram Ramanarayanan},
  title = {{Accessible Real-time Eye-gaze Tracking for Neurocognitive Health Assessment: A Multimodal Web-based Approach}},
  booktitle = {Interspeech},
  year = {2025},
}
Figure