Predicting Confusion in Information Visualization from Eye Tracking and Interaction Data

Sébastien Lallé, Cristina Conati, Giuseppe CareniniIJCAI 2016

Introduction

Information Visualization (InfoVis) tools are essential for interpreting complex data, yet they frequently induce "epistemic confusion"—a state of frustration stemming from the interface, the task, or the data itself. Research indicates that this confusion significantly impairs task accuracy and increases completion time. To address this, researchers at the University of British Columbia developed a machine learning model capable of predicting real-time confusion by monitoring user behavior.

Methodology

The study involved 136 participants performing 40 decision-making tasks using ValueChart, an interactive tool for multi-criteria preferential choice. Users performed tasks such as selecting a rental home based on various attributes like price, location, and size.

The ValueChart interface used in the study, featuring the "I am confused" button for self-reporting.

Data Collection and Feature Engineering

A multi-modal approach was used to capture user states via a Tobii T120 eye tracker and mouse logs. Four primary feature categories were derived:

  • Gaze: Fixation rates and saccade lengths.
  • Pupil: Percentage change in pupil size, serving as a proxy for cognitive load.
  • Head Distance: Fluctuations in distance from the screen, indicating engagement or posture.
  • Mouse Events: Click rates and timing.

Classification Model

The researchers employed a Random Forest classifier. Due to the extreme class imbalance—confusion occurred in only 2% of the 5,440 total trials—the SMOTE algorithm was used to synthetically over-sample confusion instances during training.

Key Insights

The study revealed several critical behavioral markers of confusion:

  • Cognitive Load: Confused users exhibited higher pupil dilation and variability, consistent with increased cognitive effort.
  • Physical Engagement: Fluctuations in head distance suggested users "lean in" to resolve ambiguity before potentially disengaging.
  • Visual Attention: Gaze data showed that confused users focused more on text labels than visual data, suggesting that ambiguous wording is a major trigger for confusion.
  • Layout Independence: The most powerful predictors—pupil size and head distance—are layout-independent, suggesting these models could generalize across different software interfaces.

Results

The Random Forest model achieved high predictive performance. Using all features, the model reached a sensitivity of 61% and a specificity of 92.6% (a false alarm rate of only 7.4%).

Classification performance (Sensitivity and Specificity) across different SMOTE levels and feature sets.

Even when restricted to pupil and head distance data—features captureable by a standard webcam—the model maintained a sensitivity of 57%. Statistical analysis (MANOVA) confirmed that confusion trials resulted in significantly lower accuracy and longer completion times, validating confusion as a major bottleneck in data analysis.

Limitations and Future Work

While promising, the model relies on self-reported "ground truth," which may be incomplete if users are confused but do not click the button. Additionally, while pupil and head data are layout-independent, gaze features remain specific to the ValueChart interface. Future research will focus on testing these models across diverse visualization styles and developing automated system interventions to assist confused users in real-time.

Reference: Lallé, S., Conati, C., & Carenini, G. (2016). Predicting Confusion in Information Visualization from Eye Tracking and Interaction Data. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16).

Summary updated

BibTeX
@inproceedings{lalle2016predicting,
  author = {Sébastien Lallé and Cristina Conati and Giuseppe Carenini},
  title = {{Predicting Confusion in Information Visualization from Eye Tracking and Interaction Data}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure