Investigating Methods for Cognitive Workload Estimation for Assistive Robots

Ayca Aygun, Thuan Nguyen, Zachary Haga, Shuchin Aeron, Matthias ScheutzSensors 2022 · OpenReview

TL;DR

This paper investigates which physiological signals are best for estimating human cognitive workload in human-robot interaction. They found that eye gaze, specifically pupil diameter, is the most effective single indicator, even outperforming combinations of signals like EEG. This is crucial for developing assistive robots that can adapt to user needs without causing overload.

What Problem Does This Paper Address?

Assistive robots are increasingly being deployed in various settings, from factories to homes. For these robots to be truly helpful, they need to understand the cognitive state of the humans they interact with. If a human is already busy and experiencing high cognitive workload, a robot shouldn't add to their burden with unnecessary interactions or tasks. The challenge lies in reliably and efficiently measuring this cognitive workload in real-time. While various physiological signals (like EEG, heart rate, and eye gaze) and machine learning models exist, it's unclear which combination offers the best performance, especially in complex, multi-modal scenarios. This paper aims to bridge this gap by systematically evaluating different physiological signals and machine learning approaches for cognitive workload estimation.

Methods

The researchers designed a simulated driving experiment to induce varying levels of cognitive workload. Participants performed a primary driving task while also engaging in secondary tasks like dialogue interactions and braking events. They collected data from multiple physiological sensors:

  • Eye Gaze (Pupillometry): Measured pupil dilation, fixation count, and fixation duration.
  • Electroencephalography (EEG): Recorded brain activity.
  • Arterial Blood Pressure (ABP): Captured heart rate variability (HRV) and blood pressure variability (BPV).

They then employed a suite of five machine learning models to classify cognitive workload levels:

  • K-Nearest Neighbor (k-NN)
  • Naive Bayes (NB)
  • Random Forest (RF)
  • Support-Vector Machines (SVM)
  • Neural Network-Based Models (NNM), including MLP, EEGNet, MLSTM-FCN, and InceptionTime.

The study explored both single-modality (using one signal type) and multi-modality (combining signals) learning approaches. For multi-modality, they investigated data-level fusion (combining raw time-series data) and feature-level fusion (concatenating extracted features).

Datasets

The study utilized data collected from 80 healthy volunteers participating in a simulated driving experiment. This experiment generated a dataset with three distinct cognitive workload levels:

  • Level 0: Baseline driving without additional tasks.
  • Level 1: Driving with dialogue tasks (answering questions).
  • Level 2: Driving with simultaneous dialogue and braking events.

The dataset was balanced, with 116 samples per workload level, totaling 348 samples. The researchers also specifically manipulated the "stimulus onset asynchrony" (SOA) between dialogue and braking events to create nuanced workload conditions, focusing on SOAs of 0, 0.5, and 1 second for Level 2.

Key Insights and Ideas

  • Eye Gaze Reigns Supreme: The most striking finding is that eye gaze, particularly pupil diameter changes (PCPS), consistently outperformed other physiological signals and even multi-modal combinations in classifying cognitive workload. This suggests that subtle changes in pupil size are highly indicative of mental effort.
  • Simplicity Trumps Complexity: Despite the allure of multi-modal fusion, the study found that combining signals like EEG with eye gaze did not significantly improve classification accuracy over using eye gaze alone. In some cases, it even slightly decreased performance. This highlights the principle of parsimony in feature selection.
  • Practicality Matters: The paper emphasizes that eye gaze is easier to collect and process than EEG, and less susceptible to noise artifacts. This makes it a highly practical choice for real-world assistive robot applications.
  • Workload Construction Nuances: The lower accuracy in distinguishing between Level 1 and Level 2 workload tasks suggests that the way workload levels are constructed can significantly impact classification performance. The specific design of Level 2, while adding braking events, might not have created a sufficiently distinct workload increase from Level 1.

Key Results

  • Highest Single-Modality Accuracy: The highest binary workload classification accuracy achieved was 80.45 ± 3.15% using Support-Vector Machines (SVM) with eye gaze (PCPS) for the 0-2 task.
  • Multi-Modality Performance: Combining eye gaze and EEG yielded a highest accuracy of 77.08 ± 3.22% with a neural network model for the 0-2 task, which was lower than the best single-modality eye gaze result.
  • APCPS Superiority: Across various machine learning models and tasks, the Percentage Change in Pupil Size (APCPS) consistently showed higher classification accuracies compared to EEG, HRV, BPV, or fixation parameters.
  • Task 1-2 Difficulty: The classification accuracy for distinguishing between workload levels 1 and 2 was generally lower across all methods, attributed to the experimental design.

Limitations and Future Work

  • Offline Determination: The current study focused on offline determination of cognitive workload. Future work should explore real-time assessment to enable adaptive robot behavior.
  • Limited Scope of Cognitive States: The study primarily focused on cognitive workload. Future research could investigate the detection of other cognitive states like mind wandering or urgency.
  • Demographic Limitations: The participant pool consisted mainly of young undergraduate/graduate students. Generalizing these findings to older populations or other demographic groups requires further investigation.
  • Future Directions: The authors suggest exploring real-time workload adaptation in human-robot interaction and investigating other cognitive states.

Summary updated

Abstract

Robots interacting with humans in assistive contexts have to be sensitive to human cognitive states to be able to provide help when it is needed and not overburden the human when the human is busy. Yet, it is currently still unclear which sensing modality might allow robots to derive the best evidence of human workload. In this work, we analyzed and modeled data from a multi-modal simulated driving study specifically designed to evaluate different levels of cognitive workload induced by various secondary tasks such as dialogue interactions and braking events in addition to the primary driving task. Specifically, we performed statistical analyses of various physiological signals including eye gaze, electroencephalography, and arterial blood pressure from the healthy volunteers and utilized several machine learning methodologies including k-nearest neighbor, naive Bayes, random forest, support-vector machines, and neural network-based models to infer human cognitive workload levels. Our analyses provide evidence for eye gaze being the best physiological indicator of human cognitive workload, even when multiple signals are combined. Specifically, the highest accuracy (in %) of binary workload classification based on eye gaze signals is 80.45 ∓ 3.15 achieved by using support-vector machines, while the highest accuracy combining eye gaze and electroencephalography is only 77.08 ∓ 3.22 achieved by a neural network-based model. Our findings are important for future efforts of real-time workload estimation in the multimodal human-robot interactive systems given that eye gaze is easy to collect and process and less susceptible to noise artifacts compared to other physiological signal modalities.
BibTeX
@inproceedings{aygun2022investigating,
  author = {Ayca Aygun and Thuan Nguyen and Zachary Haga and Shuchin Aeron and Matthias Scheutz},
  title = {{Investigating Methods for Cognitive Workload Estimation for Assistive Robots}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=o5eHNFaqVO},
}
Figure