Further Results on Predicting Cognitive Abilities for Adaptive Visualizations

Cristina Conati, Sébastien Lallé, Md. Abed Rahman, Dereck TokerIJCAI 2017 · DOI

TL;DR

This paper explores predicting users' cognitive abilities (like perceptual speed and working memory) in real-time using eye-tracking data while they interact with a complex visualization system. They demonstrate that this prediction is feasible even with noisy eye-tracking data, paving the way for more personalized and adaptive visualization experiences.

What Problem Does This Paper Address?

Information visualization (InfoVis) systems are powerful tools for analyzing complex data, but their effectiveness can be significantly influenced by a user's individual cognitive abilities. For instance, how quickly someone can process visual information or how much information they can hold in their working memory can impact their ability to understand a visualization. To create truly "adaptive" visualizations that cater to each user's needs, it's crucial to model these cognitive abilities in real-time. Previous work has shown promise in predicting some abilities (perceptual speed, visual working memory) using eye-tracking data, but this was often done with simplified visualizations and tasks. This paper aims to bridge this gap by:

  1. Extending the types of visualizations and cognitive abilities studied: They use a more realistic, complex visualization system (MetroQuest, for urban planning) and investigate additional cognitive abilities (spatial memory, visual scanning, visualization literacy).
  2. Evaluating the impact of data quality: Eye-tracking data can be notoriously noisy due to factors like user physiology or movement. This paper investigates whether predictions can still be made reliably even when the data isn't perfectly clean, which is a critical step for real-world applicability.

Methods

The researchers employed a Random Forest classifier to predict users' cognitive abilities. They used eye-tracking data collected from participants interacting with the MetroQuest system.

Key methodological aspects include:

  • Data Windowing: To simulate real-time prediction, they created ten "data windows" representing incremental percentages (10% to 100%) of the user's interaction time. This allowed them to see how early in the interaction predictions could be made.
  • Feature Engineering: They extracted a comprehensive set of features from the eye-tracking data, including:
  • Gaze Features: Derived from fixations and saccades (e.g., fixation rate, saccade length, mean/std. deviation of durations). These were computed both globally and for specific "Areas of Interest" (AOIs) within the visualization (e.g., the deviation chart, the map).
  • Pupil Features: Adjusted pupil size, capturing fluctuations over time.
  • Head Distance Features: Similar statistics to pupil features, reflecting head position relative to the screen.
  • Classification Algorithms: They compared Random Forest (RF) and Boosted Logistic Regression (LB) against a majority-class baseline. RF and LB were chosen based on prior work showing their effectiveness in similar prediction tasks.
  • Data Validity Thresholds: To assess the impact of data quality, they evaluated classifiers using datasets filtered by different "validity thresholds" (e.g., requiring 60%, 80%, or 90% of gaze samples to be valid).

Datasets

The study utilized a dataset collected from a user study involving 166 participants interacting with the MetroQuest (MQ) system. MQ is a visualization-based system designed to help users understand and make decisions about urban planning scenarios.

The cognitive abilities predicted were:

  • Perceptual Speed (PS)
  • Visual Working Memory (VisWM)
  • Spatial Memory (SpM)
  • Visual Scanning (VisScan)
  • Visualization Literacy (VisLit)

The paper does not introduce a new benchmark dataset but rather uses data from a prior study ([Lallé et al. 2017]) to extend previous findings.

Key Insights and Ideas

  • Generalizability of Eye-Tracking for User Modeling: The findings suggest that predicting cognitive abilities from eye-tracking data is not limited to simple charts but extends to more complex, realistic visualizations like those in MetroQuest. This is a significant step towards making adaptive visualizations more broadly applicable.
  • Robustness to Noisy Data: Perhaps the most impactful finding is that noisy eye-tracking data can still yield accurate predictions, especially for abilities like perceptual speed, visual scanning, and spatial memory. This is crucial because real-world eye-tracking data is rarely perfect, and this suggests that researchers might not need to discard as much data or implement overly complex cleaning procedures.
  • Early Prediction is Possible: For some cognitive abilities (like Perceptual Speed), accurate predictions can be made very early in the user's interaction (as little as 10% of the interaction time). This early detection is vital for providing timely adaptive support.
  • Trade-off Between Data Quantity and Quality: For most abilities, a larger dataset with slightly noisier data (lower validity threshold) led to better prediction accuracy than a smaller, cleaner dataset. This highlights the importance of having sufficient training data. However, for Visual Working Memory, higher data quality was more critical.

Key Results

  • Predictive Power: A Random Forest classifier significantly outperformed a majority-class baseline in predicting four out of five cognitive abilities: Perceptual Speed (PS), Visual Working Memory (VisWM), Spatial Memory (SpM), and Visual Scanning (VisScan). Visualization Literacy (VisLit) was not reliably predictable with this method.
  • Impact of Data Validity:
    • For PS, VisScan, and VisWM, a validity threshold of 0.6 (allowing more, albeit noisier, data) or 0.8 generally yielded better or comparable accuracy compared to a stricter threshold of 0.9.
  • For SpM, the validity threshold had no significant effect on prediction accuracy.
  • VisWM was the exception, where a high validity threshold (0.9) was necessary for the best predictions.
  • Early Prediction: For PS, accurate predictions were achievable as early as 10% into the interaction, with an accuracy of around 63.9%. Similar early prediction windows were found for VisScan (30% interaction) and VisWM (30% interaction).
  • Accuracy Levels: Achieved accuracies for the best-performing classifiers and optimal windows ranged from approximately 63.9% to 68.2%.

Limitations and Future Work

  • VisLit Prediction: The study was unable to reliably predict Visualization Literacy using the current eye-tracking features.
  • VisWM Data Quality: Visual Working Memory prediction was an outlier, requiring high-quality data. The authors explored this further, showing that a model trained on high-quality data could still perform reasonably well on noisier data, but with a slight drop in accuracy.
  • Future Work:
    • Investigate Noisy Data Further: Continue exploring the impact of data quality and how to leverage noisier data effectively for user modeling.
  • Realistic Settings: Expand experiments to more diverse and realistic settings to gauge the applicability of these findings.
  • Increase Performance: Explore ways to improve classifier performance, potentially by incorporating additional data sources like interaction data (e.g., button clicks, navigation patterns).
  • Adaptive Interventions: The ultimate goal is to use these predictions to build truly adaptive visualizations that dynamically adjust to user needs.

Summary updated

BibTeX
@inproceedings{conati2017further,
  author = {Cristina Conati and Sébastien Lallé and Md. Abed Rahman and Dereck Toker},
  title = {{Further Results on Predicting Cognitive Abilities for Adaptive Visualizations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/217},
}
Figure